Jul 07 19:05:47.919135 ip-10-0-141-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 19:05:47.919149 ip-10-0-141-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 19:05:47.919160 ip-10-0-141-187 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 19:05:47.919475 ip-10-0-141-187 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 19:05:59.061338 ip-10-0-141-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 19:05:59.061352 ip-10-0-141-187 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 62e3f5eb0fc04fdb950aed0164255aed --
Jul 07 19:07:59.245221 ip-10-0-141-187 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 19:07:59.580477 ip-10-0-141-187 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 19:07:59.580477 ip-10-0-141-187 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 19:07:59.580477 ip-10-0-141-187 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 19:07:59.580477 ip-10-0-141-187 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 19:07:59.580477 ip-10-0-141-187 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 19:07:59.580477 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.579813    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.583988    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584001    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584005    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584008    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584011    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584014    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584017    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584019    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584022    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584025    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584027    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584030    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584039    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584041    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584044    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584047    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584049    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584052    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584054    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:07:59.587406 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584057    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584060    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584062    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584065    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584068    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584071    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584074    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584076    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584079    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584081    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584084    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584086    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584089    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584092    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584095    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584098    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584101    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584104    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584109    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:07:59.592639 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584113    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584116    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584120    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584123    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584126    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584129    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584132    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584135    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584137    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584140    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584142    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584145    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584147    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584150    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584152    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584155    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584157    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584161    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584163    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:07:59.593645 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584165    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584168    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584170    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584173    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584175    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584178    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584181    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584183    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584187    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584189    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584192    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584195    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584198    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584200    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584204    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584206    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584209    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584211    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584214    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584216    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:07:59.595450 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584218    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584221    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584223    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584226    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584228    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584230    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584233    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584235    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584238    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584241    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584244    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584246    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584249    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584252    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584254    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584257    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584259    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584262    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584265    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584267    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:07:59.596253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584270    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584272    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584972    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584979    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584982    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584985    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584988    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584992    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584994    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.584997    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585000    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585002    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585005    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585008    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585011    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585016    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585020    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585022    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585025    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:07:59.597073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585027    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585030    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585033    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585036    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585038    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585040    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585043    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585045    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585048    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585050    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585053    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585056    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585058    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585061    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585063    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585067    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585069    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585072    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585075    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585081    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:07:59.597618 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585084    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585087    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585089    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585092    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585094    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585097    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585099    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585102    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585104    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585107    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585109    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585112    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585115    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585118    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585120    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585123    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585125    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585128    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585130    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585133    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:07:59.598417 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585135    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585138    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585140    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585144    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585146    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585149    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585154    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585157    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585160    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585163    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585165    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585168    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585171    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585175    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585177    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585180    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585183    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585185    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585188    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:07:59.599000 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585190    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585196    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585199    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585202    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585204    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585207    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585209    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585212    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585215    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585217    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585220    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585224    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585227    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585230    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585233    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585235    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585239    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585241    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585244    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585247    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:07:59.599710 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585250    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585252    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585255    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585644    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585651    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585656    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585660    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585664    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585667    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585670    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585673    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585675    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585678    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585683    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585685    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585688    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585690    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585693    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585696    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:07:59.600296 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585698    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585701    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585704    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585706    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585709    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585711    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585714    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585716    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585719    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585722    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585724    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585728    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585730    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585733    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585736    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585738    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585742    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585744    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585747    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:07:59.600974 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585750    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585753    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585755    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585758    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585760    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585763    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585765    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585769    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585772    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585774    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585777    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585779    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585782    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585785    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585788    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585790    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585793    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585796    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585798    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585801    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:07:59.601507 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585803    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585806    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585808    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585810    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585813    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585815    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585818    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585820    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585823    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585825    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585829    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585831    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585833    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585836    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585839    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585841    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585844    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585846    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585849    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585853    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:07:59.602594 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585855    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585858    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585860    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585863    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585865    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585868    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585870    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585872    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585875    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585878    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585880    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585883    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585886    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585889    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585891    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585894    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585896    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585898    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585901    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585903    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:07:59.603249 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585906    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585909    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585911    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.585926    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.585999    2567 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586006    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586015    2567 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586021    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586028    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586033    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586038    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586044    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586047    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586050    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586053    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586057    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586062    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586065    2567 flags.go:64] FLAG: --cgroup-root=""
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586068    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586071    2567 flags.go:64] FLAG: --client-ca-file=""
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586073    2567 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586077    2567 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586082    2567 flags.go:64] FLAG: --cluster-domain=""
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586085    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 19:07:59.604128 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586088    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586091    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586094    2567 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586098    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586101    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586104    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586107    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586110    2567 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586113    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586116    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586119    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586122    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586126    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586129    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586132    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586135    2567 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586138    2567 flags.go:64] FLAG: --enable-server="true"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586141    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586145    2567 flags.go:64] FLAG: --event-burst="100"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586149    2567 flags.go:64] FLAG: --event-qps="50"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586152    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586155    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586158    2567 flags.go:64] FLAG: --eviction-hard=""
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586161    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 19:07:59.604776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586165    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586168    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586172    2567 flags.go:64] FLAG: --eviction-soft=""
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586174    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586177    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586180    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586183    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586186    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586189    2567 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586191    2567 flags.go:64] FLAG: --feature-gates=""
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586195    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586198    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586201    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586204    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586207    2567 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586210    2567 flags.go:64] FLAG: --help="false"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586213    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-141-187.ec2.internal"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586216    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586219    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586221    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586225    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586242    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586246    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 19:07:59.605644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586250    2567 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586255    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586260    2567 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586264    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586269    2567 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586273    2567 flags.go:64] FLAG: --kube-reserved=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586277    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586281    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586286    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586292    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586295    2567 flags.go:64] FLAG: --lock-file=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586298    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586301    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586304    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586310    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586312    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586315    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586318    2567 flags.go:64] FLAG: --logging-format="text"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586321    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586324    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586327    2567 flags.go:64] FLAG: --manifest-url=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586330    2567 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586334    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586338    2567 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586341    2567 flags.go:64] FLAG: --max-pods="110"
Jul 07 19:07:59.721052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586344    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 19:07:59.658529 ip-10-0-141-187 systemd[1]: Started Kubernetes Kubelet.
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586347    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586350    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586353    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586356    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586359    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586362    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586368    2567 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586371    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586375    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586378    2567 flags.go:64] FLAG: --pod-cidr=""
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586383    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586389    2567 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586392    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586395    2567 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586397    2567 flags.go:64] FLAG: --port="10250"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586402    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586405    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-050596e5c9540fff0"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586408    2567 flags.go:64] FLAG: --qos-reserved=""
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586411    2567 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586414    2567 flags.go:64] FLAG: --register-node="true"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586417    2567 flags.go:64] FLAG: --register-with-taints=""
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586426    2567 flags.go:64] FLAG: --registry-burst="10"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586429    2567 flags.go:64] FLAG: --registry-qps="5"
Jul 07 19:07:59.723067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586432    2567 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586435    2567 flags.go:64] FLAG: --reserved-memory=""
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586438    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586441    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586444    2567 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586447    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586450    2567 flags.go:64] FLAG: --runonce="false"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586453    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586456    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586459    2567 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586462    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586465    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586468    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586471    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586473    2567 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586476    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586479    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586482    2567 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586485    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586489    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586493    2567 flags.go:64] FLAG: --system-cgroups=""
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586495    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586501    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586503    2567 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586508    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 19:07:59.724068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586512    2567 flags.go:64] FLAG: --tls-min-version=""
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586515    2567 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586518    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586520    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586523    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586526    2567 flags.go:64] FLAG: --v="2"
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586530    2567 flags.go:64] FLAG: --version="false"
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586534    2567 flags.go:64] FLAG: --vmodule=""
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586538    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.586541    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586653    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586656    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586660    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586663    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586665    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586668    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586671    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586674    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586676    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586679    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586682    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586684    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:07:59.725073 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586687    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586689    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586692    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586694    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586697    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586702    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586704    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586707    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586710    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586713    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586716    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586719    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586721    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586723    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586726    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586729    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586731    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586734    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586736    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586739    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:07:59.725750 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586741    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586744    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586746    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586749    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586751    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586754    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586756    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586759    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586761    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586764    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586766    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586769    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586772    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586774    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586777    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586779    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586782    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586786    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586789    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586791    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:07:59.726684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586793    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586798    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586802    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586805    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586807    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586811    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586814    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586817    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586820    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586823    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586825    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586828    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586831    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586833    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586836    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586839    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586841    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586844    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:07:59.727340 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586846    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586849    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586851    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586854    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586856    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586859    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586861    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586864    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586866    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586869    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586872    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586876    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586879    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586881    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586884    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586888    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586891    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586893    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586896    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586898    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:07:59.727990 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586901    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586903    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586906    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586909    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586913    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586928    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586931    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586934    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.586937    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.587528    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.597736    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.597751    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.598405    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.598415    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598473    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:07:59.730824 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598478    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598481    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598484    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598487    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598490    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598493    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598496    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598498    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598501    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598504    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598507    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598510    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598512    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598514    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598517    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598519    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598522    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598525    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598527    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598530    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:07:59.731357 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598533    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598536    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598538    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598541    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598544    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598548    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598551    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598555    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598557    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598560    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598562    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598566    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598568    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598571    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598574    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598576    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598579    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598581    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598584    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598586    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:07:59.732034 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598589    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598592    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598595    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598598    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598601    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598603    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598606    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598608    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598611    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598613    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598616    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598618    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598623    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598627    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598630    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598633    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598636    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598639    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598641    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:07:59.732745 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598644    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598646    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598649    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598651    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598654    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598656    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598659    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598661    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598664    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598667    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598669    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598672    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598674    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598677    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598679    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598682    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598685    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598687    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598690    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598692    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:07:59.733347 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598695    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598697    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598699    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598702    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598705    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598708    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598710    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598713    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598716    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598718    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598721    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598724    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598726    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598728    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598733    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598736    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598739    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598742    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:07:59.899531 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598744    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.598749    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598854    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598859    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598862    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598865    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598868    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598870    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598873    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598876    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598878    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598882    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598885    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598887    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598890    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598892    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 19:07:59.920834 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598895    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598897    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598900    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598902    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598905    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598908    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598911    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598925    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598929    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598931    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598934    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598937    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598939    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598942    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598945    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598948    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598950    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598953    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598955    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598958    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 19:07:59.921563 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598961    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598963    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598966    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598968    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598971    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598973    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598976    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598978    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598981    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598984    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598987    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598991    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598994    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.598997    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599000    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599003    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599006    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599010    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599013    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 19:07:59.922136 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599016    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599019    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599021    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599024    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599026    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599029    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599032    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599034    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599037    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599040    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599042    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599045    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599047    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599050    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599052    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599055    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599057    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599060    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599062    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599065    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 19:08:00.136826 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599067    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599070    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599073    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599076    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599079    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599081    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599084    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599086    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599088    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599091    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599094    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599097    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599101    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599105    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599108    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599110    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599113    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599115    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599118    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 19:08:00.137875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599121    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599123    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599126    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599129    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599131    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599134    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:07:59.599138    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.599143    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.599238    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.605250    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.606646    2567 server.go:1019] "Starting client certificate rotation"
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.606746    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.606783    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 19:08:00.138591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.608877    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 19:08:00.139098 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.610080    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 19:08:00.139098 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.619585    2567 log.go:25] "Validated CRI v1 runtime API"
Jul 07 19:08:00.139098 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.621638    2567 log.go:25] "Validated CRI v1 image API"
Jul 07 19:08:00.139098 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.622385    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 19:08:00.139098 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.628985    2567 fs.go:135] Filesystem UUIDs: map[34d44a3d-0652-4fd8-ad74-16820144b2e0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a06f91b2-6eb5-4e15-b550-d0027262cf48:/dev/nvme0n1p3]
Jul 07 19:08:00.139098 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.628999    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.634278    2567 manager.go:216] Machine: {Timestamp:2026-07-07 19:07:59.633063224 +0000 UTC m=+0.284689147 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090161 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cc1e67c691487358137e04d7e43fd SystemUUID:ec2cc1e6-7c69-1487-3581-37e04d7e43fd BootID:62e3f5eb-0fc0-4fdb-950a-ed0164255aed Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5b:7d:85:e0:fb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5b:7d:85:e0:fb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:2e:38:f2:ba:7b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.634391    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.634493    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.637181    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.637206    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.638033    2567 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.638049    2567 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.638057    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.639484    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.640894    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645422    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645556    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645588    2567 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645606    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645615    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645624    2567 kubelet.go:398] "Adding apiserver pod source"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.645632    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 19:08:00.139405 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.646136    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.646177    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.646637    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.650553    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.652389    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.652422    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653559    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653573    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653579    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653585    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653593    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653599    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653607    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653612    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653619    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653624    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653632    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.653637    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.654381    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.654392    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.657544    2567 server.go:1268] "Started kubelet"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.657625    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.657638    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.657700    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.660340    2567 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.661894    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.663109    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.665732    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.670098    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.670327    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.671083    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.671101    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.671124    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.671177    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.671248    2567 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.671255    2567 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.672789    2567 factory.go:55] Registering systemd factory
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.672802    2567 factory.go:223] Registration of the systemd container factory successfully
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.673006    2567 factory.go:153] Registering CRI-O factory
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.673018    2567 factory.go:223] Registration of the crio container factory successfully
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.673058    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.673075    2567 factory.go:103] Registering Raw factory
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.673084    2567 manager.go:1196] Started watching for new ooms in manager
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.673660    2567 manager.go:349] Starting recovery of all containers
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.675161    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.675305    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 19:08:00.139989 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.675602    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.675688    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.675160    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-187.ec2.internal.18c0177d5e330444  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-187.ec2.internal,UID:ip-10-0-141-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-187.ec2.internal,},FirstTimestamp:2026-07-07 19:07:59.657468996 +0000 UTC m=+0.309094920,LastTimestamp:2026-07-07 19:07:59.657468996 +0000 UTC m=+0.309094920,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-187.ec2.internal,}"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.690305    2567 manager.go:354] Recovery completed
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.695081    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j5s7d"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.696189    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.698148    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.698176    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.698379    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.698983    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.698993    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.699010    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.699118    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.700672    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-187.ec2.internal.18c0177d609fe88f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-187.ec2.internal,UID:ip-10-0-141-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-187.ec2.internal,},FirstTimestamp:2026-07-07 19:07:59.698159759 +0000 UTC m=+0.349785685,LastTimestamp:2026-07-07 19:07:59.698159759 +0000 UTC m=+0.349785685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-187.ec2.internal,}"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701150    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701172    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701186    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701376    2567 policy_none.go:49] "None policy: Start"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701392    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701404    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.701787    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j5s7d"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.703499    2567 policy_none.go:47] "Start"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.733555    2567 manager.go:337] "Starting Device Plugin manager"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.733675    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.733687    2567 server.go:87] "Starting device plugin registration server"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.733900    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.733910    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.734049    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.734146    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.734156    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.734601    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.734636    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.834042    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.141259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.834971    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.834993    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.835003    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.835035    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.842928    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.844147    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.844185    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.844206    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.844248    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.844814    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.844846    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-187.ec2.internal\": node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.846420    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.896286    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.944362    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"]
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.944416    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.945283    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.945310    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.945319    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.948551    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.948647    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.948671    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.950204    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.950232    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.950248    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.950850    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.950879    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.950895    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.953232    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.953254    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.953950    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.953968    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:07:59.953977    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeHasSufficientPID"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.979490    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-187.ec2.internal\" not found" node="ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.986407    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-187.ec2.internal\" not found" node="ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:07:59.997002    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.072557    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b469ca5e754e40f9721a7a95476e077f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-187.ec2.internal\" (UID: \"b469ca5e754e40f9721a7a95476e077f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.142320 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.072583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f384e1e579d8d76d06f816cd9f7c1c43-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal\" (UID: \"f384e1e579d8d76d06f816cd9f7c1c43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.143287 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.072600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f384e1e579d8d76d06f816cd9f7c1c43-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal\" (UID: \"f384e1e579d8d76d06f816cd9f7c1c43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.143287 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.097648    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.172825 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.172802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f384e1e579d8d76d06f816cd9f7c1c43-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal\" (UID: \"f384e1e579d8d76d06f816cd9f7c1c43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.172905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.172833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f384e1e579d8d76d06f816cd9f7c1c43-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal\" (UID: \"f384e1e579d8d76d06f816cd9f7c1c43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.172905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.172851    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b469ca5e754e40f9721a7a95476e077f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-187.ec2.internal\" (UID: \"b469ca5e754e40f9721a7a95476e077f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.172905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.172889    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f384e1e579d8d76d06f816cd9f7c1c43-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal\" (UID: \"f384e1e579d8d76d06f816cd9f7c1c43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.172905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.172895    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f384e1e579d8d76d06f816cd9f7c1c43-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal\" (UID: \"f384e1e579d8d76d06f816cd9f7c1c43\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.173047 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.172956    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b469ca5e754e40f9721a7a95476e077f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-187.ec2.internal\" (UID: \"b469ca5e754e40f9721a7a95476e077f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.198880 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.198853    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.283482 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.283448    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.290080 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.290060    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:00.299220 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.299202    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.401111 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.401079    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.501872 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.501845    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.603884 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.603856    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.606068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.606046    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 19:08:00.606232 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.606201    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 19:08:00.704432 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.704401    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.705598 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.705569    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 19:02:59 +0000 UTC" deadline="2028-02-14 00:29:25.019459399 +0000 UTC"
Jul 07 19:08:00.705647 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.705599    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14069h21m24.313864044s"
Jul 07 19:08:00.793126 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:00.793069    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:08:00.804543 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.804514    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:00.905238 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:00.905211    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:01.006072 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.006048    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:01.014066 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:01.014036    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb469ca5e754e40f9721a7a95476e077f.slice/crio-659a6e086509cddd791d8f118bf7a3c13629fcf5c5e068ffc82d9a3b95f3ee9c WatchSource:0}: Error finding container 659a6e086509cddd791d8f118bf7a3c13629fcf5c5e068ffc82d9a3b95f3ee9c: Status 404 returned error can't find the container with id 659a6e086509cddd791d8f118bf7a3c13629fcf5c5e068ffc82d9a3b95f3ee9c
Jul 07 19:08:01.019511 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.019487    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:08:01.026668 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:01.026640    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf384e1e579d8d76d06f816cd9f7c1c43.slice/crio-9af8c18be7cacc50ec1ab95afb82e935054219634a5ab4c113fa71e5c8a50f50 WatchSource:0}: Error finding container 9af8c18be7cacc50ec1ab95afb82e935054219634a5ab4c113fa71e5c8a50f50: Status 404 returned error can't find the container with id 9af8c18be7cacc50ec1ab95afb82e935054219634a5ab4c113fa71e5c8a50f50
Jul 07 19:08:01.106235 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.106160    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:01.208198 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.208151    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-187.ec2.internal\" not found"
Jul 07 19:08:01.233100 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.233081    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:08:01.257748 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.257728    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:08:01.271816 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.271795    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:01.287494 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.287471    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 19:08:01.288365 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.288350    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal"
Jul 07 19:08:01.295829 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.295811    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 19:08:01.501796 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.501769    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 19:08:01.647999 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.647951    2567 apiserver.go:52] "Watching apiserver"
Jul 07 19:08:01.655377 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.655351    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 19:08:01.656174 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.656152    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-jc6tj","openshift-multus/network-metrics-daemon-lg54c","openshift-network-diagnostics/network-check-target-pksgb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8","openshift-cluster-node-tuning-operator/tuned-zk2nf","openshift-dns/node-resolver-6kfr4","openshift-image-registry/node-ca-5hw59","openshift-network-operator/iptables-alerter-l9s9g","openshift-ovn-kubernetes/ovnkube-node-47hd8","kube-system/global-pull-secret-syncer-cwklg","kube-system/konnectivity-agent-zkc97","kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal","openshift-multus/multus-additional-cni-plugins-mxgpb"]
Jul 07 19:08:01.659061 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.659044    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.662022 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.661898    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vwxl7\""
Jul 07 19:08:01.662143 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.662021    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.662143 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.662105    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 19:08:01.662272 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.662178    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.662345 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.662322    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.664866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.664842    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w9s8r\""
Jul 07 19:08:01.664977 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.664893    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 19:08:01.665056 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.664996    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:01.665121 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.665064    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:01.665181 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.665142    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 19:08:01.667417 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.667402    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.669747 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.669728    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.669887 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.669753    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7xzxd\""
Jul 07 19:08:01.669887 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.669801    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.669887 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.669810    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 19:08:01.670086 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.669898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.671691 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.671673    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 19:08:01.671984 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.671962    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.672233 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.672212    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.672311 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.672217    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.672372 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.672351    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7tkb9\""
Jul 07 19:08:01.674285 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.674267    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.674524 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.674504    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.674612 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.674575    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.674612 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.674604    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5sljq\""
Jul 07 19:08:01.677090 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.677667 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677643    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 19:08:01.677764 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677672    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v4kk8\""
Jul 07 19:08:01.677764 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677647    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.677764 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677651    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 19:08:01.677764 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677644    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 19:08:01.677986 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.677961    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.678119 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.678047    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 19:08:01.679349 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679327    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.679465 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679449    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 19:08:01.679519 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679467    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 19:08:01.679519 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679507    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tqfjj\""
Jul 07 19:08:01.679746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679729    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:01.679805 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679746    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 19:08:01.679805 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.679792    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:01.679915 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.679834    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.682350 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.682330    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.682430 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.682397    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:01.683365 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683348    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 19:08:01.683494 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683477    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8700ea39-999a-468b-b53e-5cabbcc4b5ed-serviceca\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.683539 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683512    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-sys-fs\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.683539 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-var-lib-kubelet\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.683630 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683546    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:01.683630 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683561    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-socket-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.683630 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683582    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysconfig\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.683630 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b3914eb-b48e-4af2-a65e-0d759c315c29-agent-certs\") pod \"konnectivity-agent-zkc97\" (UID: \"0b3914eb-b48e-4af2-a65e-0d759c315c29\") " pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683649    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-registration-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-device-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysctl-d\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683720    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-sys\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683742    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjw9k\" (UniqueName: \"kubernetes.io/projected/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-kube-api-access-jjw9k\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683777    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-modprobe-d\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.683806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683796    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-run\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4acae846-0bb0-41fa-af29-3ec6eeb83eee-hosts-file\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5nx4\" (UniqueName: \"kubernetes.io/projected/4acae846-0bb0-41fa-af29-3ec6eeb83eee-kube-api-access-l5nx4\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b3914eb-b48e-4af2-a65e-0d759c315c29-konnectivity-ca\") pod \"konnectivity-agent-zkc97\" (UID: \"0b3914eb-b48e-4af2-a65e-0d759c315c29\") " pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzqgd\" (UniqueName: \"kubernetes.io/projected/1be29b64-3d84-40e6-ab71-faec2230c19e-kube-api-access-kzqgd\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683947    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysctl-conf\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683964    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-systemd\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.683986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-tmp\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684009    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4pk\" (UniqueName: \"kubernetes.io/projected/8700ea39-999a-468b-b53e-5cabbcc4b5ed-kube-api-access-ts4pk\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684023    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-host\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684038    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8700ea39-999a-468b-b53e-5cabbcc4b5ed-host\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684051    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-kubernetes\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-tuned\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-lib-modules\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.684571 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684148    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4acae846-0bb0-41fa-af29-3ec6eeb83eee-tmp-dir\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.684980 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.684962    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.687027 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.687007    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 19:08:01.687102 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.687068    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bpcqm\""
Jul 07 19:08:01.687573 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.687463    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.689684 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.689664    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 19:08:01.689962 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.689944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sw7c8\""
Jul 07 19:08:01.690083 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.689952    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 19:08:01.690884 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.690866    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 19:08:01.718694 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.718674    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nbm5v"
Jul 07 19:08:01.728590 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.728575    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nbm5v"
Jul 07 19:08:01.771978 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.771902    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 19:08:01.785391 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-var-lib-kubelet\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.785464 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785400    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-slash\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.785464 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785427    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-node-log\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.785464 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785454    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-log-socket\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-ovnkube-config\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785495    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-var-lib-kubelet\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785507    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c245402b-60e0-45ce-9efa-33dc232d4698-multus-daemon-config\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785566    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-socket-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785583    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-var-lib-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785601    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-os-release\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.785622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xz97\" (UniqueName: \"kubernetes.io/projected/c245402b-60e0-45ce-9efa-33dc232d4698-kube-api-access-8xz97\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785643    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d53069b6-fb41-48da-a145-8dfbe4dea760-host-slash\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-os-release\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b3914eb-b48e-4af2-a65e-0d759c315c29-agent-certs\") pod \"konnectivity-agent-zkc97\" (UID: \"0b3914eb-b48e-4af2-a65e-0d759c315c29\") " pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-registration-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-device-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785743    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-socket-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785821    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-registration-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785876    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-device-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-sys\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785948    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjw9k\" (UniqueName: \"kubernetes.io/projected/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-kube-api-access-jjw9k\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785968    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-cni-multus\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.785989    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786005    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-sys\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786051    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-etc-kubernetes\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786083    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786102    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786117    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9b59752-23f5-462c-a926-8716ec8155d2-dbus\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysctl-conf\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786174    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-systemd\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786194    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-tmp\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786216    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-systemd-units\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786238    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-run-ovn-kubernetes\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786257    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-systemd\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-netns\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786318    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-multus-certs\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786346    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysctl-conf\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786394    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-system-cni-dir\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786461    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-host\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786501    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-cnibin\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.786659 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786509    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-host\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-cnibin\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786635    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4acae846-0bb0-41fa-af29-3ec6eeb83eee-tmp-dir\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-ovnkube-script-lib\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-cni-bin\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-hostroot\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786847    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8700ea39-999a-468b-b53e-5cabbcc4b5ed-serviceca\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-sys-fs\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786955    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-systemd\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.786986    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmht\" (UniqueName: \"kubernetes.io/projected/86bce9f5-a068-47a9-bbdd-ef0784cf3415-kube-api-access-rqmht\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787007    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4acae846-0bb0-41fa-af29-3ec6eeb83eee-tmp-dir\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysconfig\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787109    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-conf-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787139    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysctl-d\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.787540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787167    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-sys-fs\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-kubelet\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787321    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysctl-d\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787392    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-sysconfig\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9b59752-23f5-462c-a926-8716ec8155d2-kubelet-config\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787440    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-modprobe-d\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787563    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-modprobe-d\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787608    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-run\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787647    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4acae846-0bb0-41fa-af29-3ec6eeb83eee-hosts-file\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787657    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8700ea39-999a-468b-b53e-5cabbcc4b5ed-serviceca\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5nx4\" (UniqueName: \"kubernetes.io/projected/4acae846-0bb0-41fa-af29-3ec6eeb83eee-kube-api-access-l5nx4\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787773    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4acae846-0bb0-41fa-af29-3ec6eeb83eee-hosts-file\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787787    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-run\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-ovn\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-cni-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.788425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-k8s-cni-cncf-io\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787895    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b3914eb-b48e-4af2-a65e-0d759c315c29-konnectivity-ca\") pod \"konnectivity-agent-zkc97\" (UID: \"0b3914eb-b48e-4af2-a65e-0d759c315c29\") " pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.787924    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzqgd\" (UniqueName: \"kubernetes.io/projected/1be29b64-3d84-40e6-ab71-faec2230c19e-kube-api-access-kzqgd\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788041    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1be29b64-3d84-40e6-ab71-faec2230c19e-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788068    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-etc-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788105    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-cni-netd\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788133    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwb5m\" (UniqueName: \"kubernetes.io/projected/9b413681-a928-4b3c-a6a5-f14435739728-kube-api-access-bwb5m\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788161    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-kubelet\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788192    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4pk\" (UniqueName: \"kubernetes.io/projected/8700ea39-999a-468b-b53e-5cabbcc4b5ed-kube-api-access-ts4pk\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788219    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c245402b-60e0-45ce-9efa-33dc232d4698-cni-binary-copy\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788248    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8700ea39-999a-468b-b53e-5cabbcc4b5ed-host\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-kubernetes\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-tuned\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788365    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-cni-bin\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788390    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-system-cni-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d53069b6-fb41-48da-a145-8dfbe4dea760-iptables-alerter-script\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.789263 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788441    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788466    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b3914eb-b48e-4af2-a65e-0d759c315c29-konnectivity-ca\") pod \"konnectivity-agent-zkc97\" (UID: \"0b3914eb-b48e-4af2-a65e-0d759c315c29\") " pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-lib-modules\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788515    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-run-netns\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788551    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-socket-dir-parent\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788569    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnz7z\" (UniqueName: \"kubernetes.io/projected/d53069b6-fb41-48da-a145-8dfbe4dea760-kube-api-access-pnz7z\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788580    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-lib-modules\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788586    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68qs\" (UniqueName: \"kubernetes.io/projected/615748c0-51f6-4d14-8991-4761f31ea68a-kube-api-access-h68qs\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788607    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-env-overrides\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788624    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b413681-a928-4b3c-a6a5-f14435739728-ovn-node-metrics-cert\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788712    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-kubernetes\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.790036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.788728    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8700ea39-999a-468b-b53e-5cabbcc4b5ed-host\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.790623 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.790096    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-tmp\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.790623 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.790430    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b3914eb-b48e-4af2-a65e-0d759c315c29-agent-certs\") pod \"konnectivity-agent-zkc97\" (UID: \"0b3914eb-b48e-4af2-a65e-0d759c315c29\") " pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.790833 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.790805    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-etc-tuned\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.794635 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.794613    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:01.794750 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.794644    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:01.794750 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.794658    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:01.794862 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.794793    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:02.294714188 +0000 UTC m=+2.946340130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:01.797374 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.797349    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5nx4\" (UniqueName: \"kubernetes.io/projected/4acae846-0bb0-41fa-af29-3ec6eeb83eee-kube-api-access-l5nx4\") pod \"node-resolver-6kfr4\" (UID: \"4acae846-0bb0-41fa-af29-3ec6eeb83eee\") " pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.797499 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.797358    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4pk\" (UniqueName: \"kubernetes.io/projected/8700ea39-999a-468b-b53e-5cabbcc4b5ed-kube-api-access-ts4pk\") pod \"node-ca-5hw59\" (UID: \"8700ea39-999a-468b-b53e-5cabbcc4b5ed\") " pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.799571 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.799545    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzqgd\" (UniqueName: \"kubernetes.io/projected/1be29b64-3d84-40e6-ab71-faec2230c19e-kube-api-access-kzqgd\") pod \"aws-ebs-csi-driver-node-b8jb8\" (UID: \"1be29b64-3d84-40e6-ab71-faec2230c19e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.799683 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.799650    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjw9k\" (UniqueName: \"kubernetes.io/projected/7c7b4a76-0d98-4e14-bf9e-2657b8f0b307-kube-api-access-jjw9k\") pod \"tuned-zk2nf\" (UID: \"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307\") " pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.852628 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.852563    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal" event={"ID":"b469ca5e754e40f9721a7a95476e077f","Type":"ContainerStarted","Data":"659a6e086509cddd791d8f118bf7a3c13629fcf5c5e068ffc82d9a3b95f3ee9c"}
Jul 07 19:08:01.854418 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.854389    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal" event={"ID":"f384e1e579d8d76d06f816cd9f7c1c43","Type":"ContainerStarted","Data":"9af8c18be7cacc50ec1ab95afb82e935054219634a5ab4c113fa71e5c8a50f50"}
Jul 07 19:08:01.889292 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889264    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-env-overrides\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889478 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889304    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b413681-a928-4b3c-a6a5-f14435739728-ovn-node-metrics-cert\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889478 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-slash\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889478 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889386    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-slash\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889478 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889444    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-node-log\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889478 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-log-socket\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-ovnkube-config\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889505    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-node-log\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889527    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c245402b-60e0-45ce-9efa-33dc232d4698-multus-daemon-config\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889555    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-log-socket\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-var-lib-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-os-release\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xz97\" (UniqueName: \"kubernetes.io/projected/c245402b-60e0-45ce-9efa-33dc232d4698-kube-api-access-8xz97\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889635    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-var-lib-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889654    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d53069b6-fb41-48da-a145-8dfbe4dea760-host-slash\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-os-release\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-cni-multus\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889731    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-os-release\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.889759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889738    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-etc-kubernetes\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889867    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9b59752-23f5-462c-a926-8716ec8155d2-dbus\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889884    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-systemd-units\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889921    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-run-ovn-kubernetes\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889930    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d53069b6-fb41-48da-a145-8dfbe4dea760-host-slash\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-netns\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.889986    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-multus-certs\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-system-cni-dir\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890042    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-cnibin\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890052    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-cni-multus\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890070    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-cnibin\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890104    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-ovnkube-script-lib\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.890397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890165    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-cni-bin\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890206    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d9b59752-23f5-462c-a926-8716ec8155d2-dbus\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890234    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-cnibin\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890225    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-systemd-units\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890285    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c245402b-60e0-45ce-9efa-33dc232d4698-multus-daemon-config\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890288    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-ovnkube-config\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890294    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-cnibin\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890285    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890330    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-run-ovn-kubernetes\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890014    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-os-release\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890349    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-etc-kubernetes\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890357    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/615748c0-51f6-4d14-8991-4761f31ea68a-system-cni-dir\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890372    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-hostroot\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890393    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-cni-bin\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890399    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-netns\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890402    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890431    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-systemd\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890437    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-hostroot\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.891268 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890441    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-multus-certs\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890459    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmht\" (UniqueName: \"kubernetes.io/projected/86bce9f5-a068-47a9-bbdd-ef0784cf3415-kube-api-access-rqmht\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890482    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-systemd\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-conf-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890513    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-kubelet\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9b59752-23f5-462c-a926-8716ec8155d2-kubelet-config\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-ovn\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890613    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-cni-binary-copy\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-cni-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-k8s-cni-cncf-io\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890687    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-cni-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890690    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-etc-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890721    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-cni-netd\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890726    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-etc-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890729    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-run-k8s-cni-cncf-io\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwb5m\" (UniqueName: \"kubernetes.io/projected/9b413681-a928-4b3c-a6a5-f14435739728-kube-api-access-bwb5m\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890776    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-kubelet\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890780    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-kubelet\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890793    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-ovnkube-script-lib\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890803    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c245402b-60e0-45ce-9efa-33dc232d4698-cni-binary-copy\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890803    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d9b59752-23f5-462c-a926-8716ec8155d2-kubelet-config\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890823    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-conf-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890834    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-ovn\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890852    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-cni-bin\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890856    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-host-var-lib-kubelet\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890873    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-cni-netd\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890890    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9b413681-a928-4b3c-a6a5-f14435739728-env-overrides\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890903    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-cni-bin\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.890897    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890911    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-system-cni-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.890942    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d53069b6-fb41-48da-a145-8dfbe4dea760-iptables-alerter-script\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.890985    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:02.390965928 +0000 UTC m=+3.042591859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891027    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:01.892983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891075    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-system-cni-dir\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891119    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-run-netns\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-socket-dir-parent\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnz7z\" (UniqueName: \"kubernetes.io/projected/d53069b6-fb41-48da-a145-8dfbe4dea760-kube-api-access-pnz7z\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68qs\" (UniqueName: \"kubernetes.io/projected/615748c0-51f6-4d14-8991-4761f31ea68a-kube-api-access-h68qs\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.891274    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891323    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-host-run-netns\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:01.891396    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:02.391370681 +0000 UTC m=+3.042996612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891435    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9b413681-a928-4b3c-a6a5-f14435739728-run-openvswitch\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891452    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891494    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/615748c0-51f6-4d14-8991-4761f31ea68a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891513    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c245402b-60e0-45ce-9efa-33dc232d4698-cni-binary-copy\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891601    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d53069b6-fb41-48da-a145-8dfbe4dea760-iptables-alerter-script\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.891677    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c245402b-60e0-45ce-9efa-33dc232d4698-multus-socket-dir-parent\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.893910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.892060    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9b413681-a928-4b3c-a6a5-f14435739728-ovn-node-metrics-cert\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.900875 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.900851    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xz97\" (UniqueName: \"kubernetes.io/projected/c245402b-60e0-45ce-9efa-33dc232d4698-kube-api-access-8xz97\") pod \"multus-jc6tj\" (UID: \"c245402b-60e0-45ce-9efa-33dc232d4698\") " pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:01.901870 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.901833    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmht\" (UniqueName: \"kubernetes.io/projected/86bce9f5-a068-47a9-bbdd-ef0784cf3415-kube-api-access-rqmht\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:01.902089 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.902064    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68qs\" (UniqueName: \"kubernetes.io/projected/615748c0-51f6-4d14-8991-4761f31ea68a-kube-api-access-h68qs\") pod \"multus-additional-cni-plugins-mxgpb\" (UID: \"615748c0-51f6-4d14-8991-4761f31ea68a\") " pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:01.902948 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.902882    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnz7z\" (UniqueName: \"kubernetes.io/projected/d53069b6-fb41-48da-a145-8dfbe4dea760-kube-api-access-pnz7z\") pod \"iptables-alerter-l9s9g\" (UID: \"d53069b6-fb41-48da-a145-8dfbe4dea760\") " pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:01.903103 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.903073    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwb5m\" (UniqueName: \"kubernetes.io/projected/9b413681-a928-4b3c-a6a5-f14435739728-kube-api-access-bwb5m\") pod \"ovnkube-node-47hd8\" (UID: \"9b413681-a928-4b3c-a6a5-f14435739728\") " pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:01.972106 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.972066    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5hw59"
Jul 07 19:08:01.977912 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.977896    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:01.984178 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:01.984153    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8700ea39_999a_468b_b53e_5cabbcc4b5ed.slice/crio-a61132300b52b0dd2541a22eb239ed5f2ae7e58f24705b34bd707528f183c9fa WatchSource:0}: Error finding container a61132300b52b0dd2541a22eb239ed5f2ae7e58f24705b34bd707528f183c9fa: Status 404 returned error can't find the container with id a61132300b52b0dd2541a22eb239ed5f2ae7e58f24705b34bd707528f183c9fa
Jul 07 19:08:01.985519 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.985500    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8"
Jul 07 19:08:01.987164 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:01.987144    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3914eb_b48e_4af2_a65e_0d759c315c29.slice/crio-4a857ba9df7dc9153f4d829715de6bc4a9d33c89c6fe0ef6400f48864a62cc68 WatchSource:0}: Error finding container 4a857ba9df7dc9153f4d829715de6bc4a9d33c89c6fe0ef6400f48864a62cc68: Status 404 returned error can't find the container with id 4a857ba9df7dc9153f4d829715de6bc4a9d33c89c6fe0ef6400f48864a62cc68
Jul 07 19:08:01.990538 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.990523    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zk2nf"
Jul 07 19:08:01.995748 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:01.995720    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be29b64_3d84_40e6_ab71_faec2230c19e.slice/crio-553bdac54bd1b350a9ffaafcfd030d59aca943efbd387fd110bb2c388cd42e38 WatchSource:0}: Error finding container 553bdac54bd1b350a9ffaafcfd030d59aca943efbd387fd110bb2c388cd42e38: Status 404 returned error can't find the container with id 553bdac54bd1b350a9ffaafcfd030d59aca943efbd387fd110bb2c388cd42e38
Jul 07 19:08:01.998968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:01.998950    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6kfr4"
Jul 07 19:08:01.999693 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:01.999673    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7b4a76_0d98_4e14_bf9e_2657b8f0b307.slice/crio-b105dda632c0853a81cb90b035c4f351271932c95bc73d15c50df3a534808d0e WatchSource:0}: Error finding container b105dda632c0853a81cb90b035c4f351271932c95bc73d15c50df3a534808d0e: Status 404 returned error can't find the container with id b105dda632c0853a81cb90b035c4f351271932c95bc73d15c50df3a534808d0e
Jul 07 19:08:02.004133 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.004115    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:02.008676 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:02.008644    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acae846_0bb0_41fa_af29_3ec6eeb83eee.slice/crio-b212c0fa96cfe67c76f5a54334e759301611af76b389d34e46930be8967d435c WatchSource:0}: Error finding container b212c0fa96cfe67c76f5a54334e759301611af76b389d34e46930be8967d435c: Status 404 returned error can't find the container with id b212c0fa96cfe67c76f5a54334e759301611af76b389d34e46930be8967d435c
Jul 07 19:08:02.010307 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.010284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mxgpb"
Jul 07 19:08:02.013102 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:02.013082    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b413681_a928_4b3c_a6a5_f14435739728.slice/crio-18895502c30242653cb3496c000f7fd353869eb9222bc1cbfa4d33c4b874b900 WatchSource:0}: Error finding container 18895502c30242653cb3496c000f7fd353869eb9222bc1cbfa4d33c4b874b900: Status 404 returned error can't find the container with id 18895502c30242653cb3496c000f7fd353869eb9222bc1cbfa4d33c4b874b900
Jul 07 19:08:02.015638 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.015621    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jc6tj"
Jul 07 19:08:02.020207 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:02.020182    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615748c0_51f6_4d14_8991_4761f31ea68a.slice/crio-84bbc1ac4eced4e18a0a6c942236b3bb86e4dbec35d8e844b073cfc1a6223672 WatchSource:0}: Error finding container 84bbc1ac4eced4e18a0a6c942236b3bb86e4dbec35d8e844b073cfc1a6223672: Status 404 returned error can't find the container with id 84bbc1ac4eced4e18a0a6c942236b3bb86e4dbec35d8e844b073cfc1a6223672
Jul 07 19:08:02.020895 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.020877    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l9s9g"
Jul 07 19:08:02.029637 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:02.029614    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc245402b_60e0_45ce_9efa_33dc232d4698.slice/crio-001eec543c282c63543b4452e4693add0ad5ff073b2535b291617c41d77cddfc WatchSource:0}: Error finding container 001eec543c282c63543b4452e4693add0ad5ff073b2535b291617c41d77cddfc: Status 404 returned error can't find the container with id 001eec543c282c63543b4452e4693add0ad5ff073b2535b291617c41d77cddfc
Jul 07 19:08:02.032896 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:02.032879    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53069b6_fb41_48da_a145_8dfbe4dea760.slice/crio-aac8f9e8c75296ee162dd4963c280284292ac82ed5bafd975eff02e010f15fc5 WatchSource:0}: Error finding container aac8f9e8c75296ee162dd4963c280284292ac82ed5bafd975eff02e010f15fc5: Status 404 returned error can't find the container with id aac8f9e8c75296ee162dd4963c280284292ac82ed5bafd975eff02e010f15fc5
Jul 07 19:08:02.394906 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.394811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:02.394906 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.394879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:02.395153 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.394918    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:02.395153 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395072    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:02.395153 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395092    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:02.395153 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395104    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:02.395364 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395167    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:03.395146216 +0000 UTC m=+4.046772129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:02.395364 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395264    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:02.395364 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395331    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:03.395311966 +0000 UTC m=+4.046937879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:02.395364 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395263    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:02.395619 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.395379    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:03.395369269 +0000 UTC m=+4.046995188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:02.732562 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.732481    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:03:01 +0000 UTC" deadline="2028-04-16 22:47:28.895775848 +0000 UTC"
Jul 07 19:08:02.732562 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.732516    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15579h39m26.163264444s"
Jul 07 19:08:02.844901 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.844870    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:02.845053 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:02.844992    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:02.860188 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.860153    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"18895502c30242653cb3496c000f7fd353869eb9222bc1cbfa4d33c4b874b900"}
Jul 07 19:08:02.861500 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.861465    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8" event={"ID":"1be29b64-3d84-40e6-ab71-faec2230c19e","Type":"ContainerStarted","Data":"553bdac54bd1b350a9ffaafcfd030d59aca943efbd387fd110bb2c388cd42e38"}
Jul 07 19:08:02.862610 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.862590    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9s9g" event={"ID":"d53069b6-fb41-48da-a145-8dfbe4dea760","Type":"ContainerStarted","Data":"aac8f9e8c75296ee162dd4963c280284292ac82ed5bafd975eff02e010f15fc5"}
Jul 07 19:08:02.864265 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.864239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jc6tj" event={"ID":"c245402b-60e0-45ce-9efa-33dc232d4698","Type":"ContainerStarted","Data":"001eec543c282c63543b4452e4693add0ad5ff073b2535b291617c41d77cddfc"}
Jul 07 19:08:02.865487 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.865441    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerStarted","Data":"84bbc1ac4eced4e18a0a6c942236b3bb86e4dbec35d8e844b073cfc1a6223672"}
Jul 07 19:08:02.866502 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.866481    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6kfr4" event={"ID":"4acae846-0bb0-41fa-af29-3ec6eeb83eee","Type":"ContainerStarted","Data":"b212c0fa96cfe67c76f5a54334e759301611af76b389d34e46930be8967d435c"}
Jul 07 19:08:02.867918 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.867892    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zk2nf" event={"ID":"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307","Type":"ContainerStarted","Data":"b105dda632c0853a81cb90b035c4f351271932c95bc73d15c50df3a534808d0e"}
Jul 07 19:08:02.869359 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.869312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkc97" event={"ID":"0b3914eb-b48e-4af2-a65e-0d759c315c29","Type":"ContainerStarted","Data":"4a857ba9df7dc9153f4d829715de6bc4a9d33c89c6fe0ef6400f48864a62cc68"}
Jul 07 19:08:02.870800 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:02.870774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5hw59" event={"ID":"8700ea39-999a-468b-b53e-5cabbcc4b5ed","Type":"ContainerStarted","Data":"a61132300b52b0dd2541a22eb239ed5f2ae7e58f24705b34bd707528f183c9fa"}
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.405209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.405363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405418    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405450    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.405452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405464    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:03.405516 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405527    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:05.405508342 +0000 UTC m=+6.057134256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:03.406052 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405686    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:03.406052 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405753    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:05.405725671 +0000 UTC m=+6.057351600 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:03.406052 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.405895    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:03.406052 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.406031    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:05.405967016 +0000 UTC m=+6.057592929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:03.734016 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.733701    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 19:03:01 +0000 UTC" deadline="2028-01-08 22:52:24.877729006 +0000 UTC"
Jul 07 19:08:03.734016 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.734015    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13203h44m21.14372081s"
Jul 07 19:08:03.844677 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.844644    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:03.847172 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.847138    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:03.847539 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.847519    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:03.847646 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:03.847627    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:03.887859 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:03.887787    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal" event={"ID":"b469ca5e754e40f9721a7a95476e077f","Type":"ContainerStarted","Data":"f7643435f0d6a3120a0b98f60fe1b2ef474a4b42325120cf168d4827a43260c8"}
Jul 07 19:08:04.844576 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:04.844546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:04.845059 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:04.844675    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:05.422971 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:05.422912    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:05.423169 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:05.423011    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:05.423169 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:05.423061    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:05.423284 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423184    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:05.423284 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423243    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:09.423224126 +0000 UTC m=+10.074850055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:05.423654 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423634    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:05.423735 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423660    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:05.423735 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423674    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:05.423735 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423726    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:09.423704755 +0000 UTC m=+10.075330669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:05.423902 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423787    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:05.423902 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.423821    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:09.423810829 +0000 UTC m=+10.075436744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:05.847697 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:05.847037    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:05.847697 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.847175    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:05.847697 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:05.847526    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:05.847697 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:05.847620    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:06.844991 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:06.844954    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:06.845169 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:06.845088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:07.845510 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:07.845425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:07.845972 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:07.845425    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:07.845972 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:07.845563    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:07.845972 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:07.845652    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:08.844744 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:08.844706    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:08.844951 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:08.844831    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:09.455662 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:09.455628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:09.455688    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:09.455742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455796    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455798    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455852    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455864    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:17.4558445 +0000 UTC m=+18.107470429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455868    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455882    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455884    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:17.455872925 +0000 UTC m=+18.107498843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:09.456097 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.455939    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:17.455913411 +0000 UTC m=+18.107539322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:09.845863 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:09.845827    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:09.846084 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.845953    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:09.846084 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:09.846023    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:09.846207 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:09.846150    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:10.844989 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:10.844946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:10.845416 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:10.845070    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:11.845071 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:11.845038    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:11.845503 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:11.845090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:11.845503 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:11.845196    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:11.845503 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:11.845293    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:12.844357 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:12.844327    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:12.844497 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:12.844447    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:13.845398 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:13.845360    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:13.845889 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:13.845416    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:13.845889 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:13.845524    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:13.845889 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:13.845597    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:13.958676 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:13.958640    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zk2nf" event={"ID":"7c7b4a76-0d98-4e14-bf9e-2657b8f0b307","Type":"ContainerStarted","Data":"e3588acce44592b6c13ca6a1582552d7e88b1ab2ddac68367bc39ff1c19ae35e"}
Jul 07 19:08:13.980350 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:13.980295    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-187.ec2.internal" podStartSLOduration=12.980276988 podStartE2EDuration="12.980276988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:03.905390766 +0000 UTC m=+4.557016700" watchObservedRunningTime="2026-07-07 19:08:13.980276988 +0000 UTC m=+14.631902922"
Jul 07 19:08:13.980820 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:13.980774    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zk2nf" podStartSLOduration=3.121467601 podStartE2EDuration="13.980763363s" totalImagesPullingTime="10.859295762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:13.980546035 +0000 UTC m=+14.632171969" watchObservedRunningTime="2026-07-07 19:08:13.980763363 +0000 UTC m=+14.632389297"
Jul 07 19:08:14.845346 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.845153    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:14.845518 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:14.845425    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:14.961527 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.961496    2567 generic.go:356] "Generic (PLEG): container finished" podID="f384e1e579d8d76d06f816cd9f7c1c43" containerID="138fe3ca75ac496e3feced8c28ac6146aa191b245b7794670b43829ac998a6d8" exitCode=0
Jul 07 19:08:14.961703 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.961583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal" event={"ID":"f384e1e579d8d76d06f816cd9f7c1c43","Type":"ContainerDied","Data":"138fe3ca75ac496e3feced8c28ac6146aa191b245b7794670b43829ac998a6d8"}
Jul 07 19:08:14.963020 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.962990    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l9s9g" event={"ID":"d53069b6-fb41-48da-a145-8dfbe4dea760","Type":"ContainerStarted","Data":"339c635c11433d77b5a10873bfa7764523a05c1fedc2f0f4a3b706ab6604b7eb"}
Jul 07 19:08:14.964457 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.964434    2567 generic.go:356] "Generic (PLEG): container finished" podID="615748c0-51f6-4d14-8991-4761f31ea68a" containerID="b2e7a4f3c799e891336b417a173d3b981c685b9451c3c037394553f8f783e716" exitCode=0
Jul 07 19:08:14.964556 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.964470    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerDied","Data":"b2e7a4f3c799e891336b417a173d3b981c685b9451c3c037394553f8f783e716"}
Jul 07 19:08:14.966411 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.966383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6kfr4" event={"ID":"4acae846-0bb0-41fa-af29-3ec6eeb83eee","Type":"ContainerStarted","Data":"6138f89efba277d07cae0e5cf734659f5961d9a66d8b517e6db939760392920c"}
Jul 07 19:08:14.967807 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.967785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkc97" event={"ID":"0b3914eb-b48e-4af2-a65e-0d759c315c29","Type":"ContainerStarted","Data":"305af5ce39ae29e5626e7497ed192a7081662d1293128108307a8583d7e1f634"}
Jul 07 19:08:14.969539 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.969517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5hw59" event={"ID":"8700ea39-999a-468b-b53e-5cabbcc4b5ed","Type":"ContainerStarted","Data":"83781c91e6b759b9ba71c1c687b6c4e59e97a53cef65495955ce7700259818ee"}
Jul 07 19:08:14.971071 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:14.971045    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8" event={"ID":"1be29b64-3d84-40e6-ab71-faec2230c19e","Type":"ContainerStarted","Data":"ac190dd9fef229188e280c602db294f79402b2d8a73afe019d382198217f4d0f"}
Jul 07 19:08:15.000785 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.000746    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5hw59" podStartSLOduration=4.142927693 podStartE2EDuration="15.000733548s" totalImagesPullingTime="10.857805855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:15.00043887 +0000 UTC m=+15.652064837" watchObservedRunningTime="2026-07-07 19:08:15.000733548 +0000 UTC m=+15.652359482"
Jul 07 19:08:15.019537 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.019507    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:15.020022 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.020000    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:15.021145 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.021101    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zkc97" podStartSLOduration=4.493431147 podStartE2EDuration="15.02108729s" totalImagesPullingTime="10.527656143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:15.02051061 +0000 UTC m=+15.672136543" watchObservedRunningTime="2026-07-07 19:08:15.02108729 +0000 UTC m=+15.672713223"
Jul 07 19:08:15.047347 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.047290    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6kfr4" podStartSLOduration=4.213847427 podStartE2EDuration="15.047271117s" totalImagesPullingTime="10.83342369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:15.046489631 +0000 UTC m=+15.698115565" watchObservedRunningTime="2026-07-07 19:08:15.047271117 +0000 UTC m=+15.698897051"
Jul 07 19:08:15.099251 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.099085    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l9s9g" podStartSLOduration=4.288250279 podStartE2EDuration="15.099064813s" totalImagesPullingTime="10.810814534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:15.09775466 +0000 UTC m=+15.749380596" watchObservedRunningTime="2026-07-07 19:08:15.099064813 +0000 UTC m=+15.750690747"
Jul 07 19:08:15.844579 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.844546    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:15.844579 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.844577    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:15.844796 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:15.844705    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:15.844883 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:15.844854    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:15.974584 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.974547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal" event={"ID":"f384e1e579d8d76d06f816cd9f7c1c43","Type":"ContainerStarted","Data":"06d9d807f5ce09ad74df5f0bf749c6bc5fceda15156ffadd35cc54386c3fbe8b"}
Jul 07 19:08:15.975281 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.975263    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:15.975801 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.975781    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zkc97"
Jul 07 19:08:15.994175 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:15.994127    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-187.ec2.internal" podStartSLOduration=14.994113333 podStartE2EDuration="14.994113333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:15.993956381 +0000 UTC m=+16.645582316" watchObservedRunningTime="2026-07-07 19:08:15.994113333 +0000 UTC m=+16.645739267"
Jul 07 19:08:16.845083 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:16.845005    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:16.845318 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:16.845151    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:17.521775 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:17.521736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:17.521796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.521895    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.521938    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.521957    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.521970    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.521972    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:33.521951826 +0000 UTC m=+34.173577748 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:17.521997    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.522020    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:33.522005093 +0000 UTC m=+34.173631019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.522052    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:17.522500 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.522101    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:33.522089277 +0000 UTC m=+34.173715188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:17.845306 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:17.845224    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:17.845306 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:17.845276    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:17.845520 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.845368    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:17.845520 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:17.845452    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:18.845241 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:18.845207    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:18.845663 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:18.845317    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:19.846033 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:19.846004    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:19.846385 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:19.846076    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:19.846385 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:19.846091    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:19.846385 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:19.846165    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:20.845351 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:20.845156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:20.845520 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:20.845441    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:21.845343 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:21.845301    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:21.845849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:21.845301    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:21.845849 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:21.845439    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:21.845849 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:21.845481    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:22.845394 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:22.845360    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:22.845801 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:22.845470    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:23.845467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:23.845432    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:23.845467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:23.845476    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:23.846075 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:23.845608    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:23.846075 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:23.845776    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:24.845221 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:24.845081    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:24.845379 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:24.845211    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:25.844601 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:25.844571    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:25.845036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:25.844578    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:25.845036 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:25.844676    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:25.845036 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:25.844791    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:26.375432 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:26.375407    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 19:08:26.749529 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:26.749115    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:08:26.375427449Z","UUID":"feb99914-30f4-4304-bfdc-a2979b07356a","Handler":null,"Name":"","Endpoint":""}
Jul 07 19:08:26.750583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:26.750563    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 19:08:26.750668 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:26.750603    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 19:08:26.844661 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:26.844641    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:26.845024 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:26.844721    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:26.998456 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:26.998423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jc6tj" event={"ID":"c245402b-60e0-45ce-9efa-33dc232d4698","Type":"ContainerStarted","Data":"2c5702a64f353e78da6a137194f0ad27f64713cf99ae467e0a0ea126abba7f53"}
Jul 07 19:08:27.000525 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.000499    2567 generic.go:356] "Generic (PLEG): container finished" podID="615748c0-51f6-4d14-8991-4761f31ea68a" containerID="40a7b670c8c297cb31827c97dd882f6ad20b46a74e2c41c0fff88e4de9346257" exitCode=0
Jul 07 19:08:27.000638 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.000577    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerDied","Data":"40a7b670c8c297cb31827c97dd882f6ad20b46a74e2c41c0fff88e4de9346257"}
Jul 07 19:08:27.004046 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004027    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:08:27.004482 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004463    2567 generic.go:356] "Generic (PLEG): container finished" podID="9b413681-a928-4b3c-a6a5-f14435739728" containerID="85f6ba7c10921de982693375167f2882f06cfb8cfe2edece7f7d3d9ae48bd480" exitCode=1
Jul 07 19:08:27.004559 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004527    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"67c670b64e38e36b9fc9e1499f02c8219157c88a9de6314d1a348f39fc0ccc41"}
Jul 07 19:08:27.004612 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"0644d2d6ea88847a6440efe515d7cba7372e97d1012d38808fe9ff4b6e9a94dd"}
Jul 07 19:08:27.004665 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004629    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"5432abc334445a0010adafc0516d6a11db808f81c6a8d46496cc1a679620338e"}
Jul 07 19:08:27.004704 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004680    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"521128ef4bae7a4ee6d30340dec5583a81aac39828f7ec5b0ffce2901aa705be"}
Jul 07 19:08:27.004745 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004726    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerDied","Data":"85f6ba7c10921de982693375167f2882f06cfb8cfe2edece7f7d3d9ae48bd480"}
Jul 07 19:08:27.004790 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.004774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"87ab312018ddd4c724bcf37cd64878dc26d85cdf3cfeb5cdb290facf55dfbd7e"}
Jul 07 19:08:27.006439 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.006416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8" event={"ID":"1be29b64-3d84-40e6-ab71-faec2230c19e","Type":"ContainerStarted","Data":"2f315df3df0c7c136fde9e623bc31b71f8d699b5f781ec5ee609b90c1e071548"}
Jul 07 19:08:27.018555 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.018510    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jc6tj" podStartSLOduration=4.362506429 podStartE2EDuration="27.018476864s" totalImagesPullingTime="22.655970435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:27.018465751 +0000 UTC m=+27.670091695" watchObservedRunningTime="2026-07-07 19:08:27.018476864 +0000 UTC m=+27.670102798"
Jul 07 19:08:27.845172 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.845139    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:27.845854 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:27.845297    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:27.845854 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:27.845338    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:27.845854 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:27.845430    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:28.010934 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:28.010889    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8" event={"ID":"1be29b64-3d84-40e6-ab71-faec2230c19e","Type":"ContainerStarted","Data":"1a41d089de3d4b657064f8949806c96848ff91300613fc65a0ea10344786db3a"}
Jul 07 19:08:28.034910 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:28.032720    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8jb8" podStartSLOduration=5.791606086 podStartE2EDuration="28.032697532s" totalImagesPullingTime="22.241091446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:28.032474736 +0000 UTC m=+28.684100670" watchObservedRunningTime="2026-07-07 19:08:28.032697532 +0000 UTC m=+28.684323466"
Jul 07 19:08:28.844489 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:28.844458    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:28.844668 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:28.844553    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:29.014888 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:29.014864    2567 generic.go:356] "Generic (PLEG): container finished" podID="615748c0-51f6-4d14-8991-4761f31ea68a" containerID="79aea46253b7566fdb6106dc65189fd2579713c442cadcd04e714b60fe00b883" exitCode=0
Jul 07 19:08:29.015260 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:29.014950    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerDied","Data":"79aea46253b7566fdb6106dc65189fd2579713c442cadcd04e714b60fe00b883"}
Jul 07 19:08:29.845499 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:29.845464    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:29.845680 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:29.845577    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:29.845680 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:29.845652    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:29.845800 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:29.845727    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:30.019484 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:30.019459    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:08:30.019975 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:30.019732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"b764c99e84a9369f38f87e8b8c77d369fa40241263ad00a2e15d003a8ed3fbf4"}
Jul 07 19:08:30.844956 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:30.844911    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:30.845117 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:30.845019    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:31.023768 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:31.023732    2567 generic.go:356] "Generic (PLEG): container finished" podID="615748c0-51f6-4d14-8991-4761f31ea68a" containerID="8eeecbd19a60d4bb229e78113681c50626e505f718685a250a2f8785bb2a3df7" exitCode=0
Jul 07 19:08:31.024144 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:31.023774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerDied","Data":"8eeecbd19a60d4bb229e78113681c50626e505f718685a250a2f8785bb2a3df7"}
Jul 07 19:08:31.845345 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:31.845168    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:31.845512 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:31.845170    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:31.845512 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:31.845420    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:31.845512 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:31.845489    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:32.029592 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.029567    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:08:32.030033 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.029949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"843da76fefb7411012a812d5fbb54cc01ee3ffadb122779784a1da59716726e4"}
Jul 07 19:08:32.030301 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.030278    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:32.030425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.030355    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:32.030502 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.030447    2567 scope.go:117] "RemoveContainer" containerID="85f6ba7c10921de982693375167f2882f06cfb8cfe2edece7f7d3d9ae48bd480"
Jul 07 19:08:32.050556 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.050536    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:32.845335 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:32.845295    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:32.845494 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:32.845409    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:33.036137 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.036074    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:08:33.036567 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.036494    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" event={"ID":"9b413681-a928-4b3c-a6a5-f14435739728","Type":"ContainerStarted","Data":"7c9289cdae6c64201c0223e1916f3185a079eba28c97ed20c2cee792cc279626"}
Jul 07 19:08:33.036757 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.036736    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:33.051870 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.051845    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:08:33.068214 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.068157    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8" podStartSLOduration=8.799593772 podStartE2EDuration="33.068136749s" totalImagesPullingTime="24.268542977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:33.066639891 +0000 UTC m=+33.718265825" watchObservedRunningTime="2026-07-07 19:08:33.068136749 +0000 UTC m=+33.719762684"
Jul 07 19:08:33.515609 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.515574    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cwklg"]
Jul 07 19:08:33.515764 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.515706    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:33.515824 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.515794    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:33.519306 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.519282    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lg54c"]
Jul 07 19:08:33.519413 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.519382    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:33.519489 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.519474    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:33.519856 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.519833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pksgb"]
Jul 07 19:08:33.519934 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.519907    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:33.520022 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.519999    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:33.538771 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.538748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:33.538884 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.538789    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:33.538884 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:33.538833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:33.539009 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.538884    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:33.539009 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.538938    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:33.539009 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.538953    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret podName:d9b59752-23f5-462c-a926-8716ec8155d2 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:05.538936751 +0000 UTC m=+66.190562674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret") pod "global-pull-secret-syncer-cwklg" (UID: "d9b59752-23f5-462c-a926-8716ec8155d2") : object "kube-system"/"original-pull-secret" not registered
Jul 07 19:08:33.539009 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.538986    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs podName:86bce9f5-a068-47a9-bbdd-ef0784cf3415 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:05.538971403 +0000 UTC m=+66.190597313 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs") pod "network-metrics-daemon-lg54c" (UID: "86bce9f5-a068-47a9-bbdd-ef0784cf3415") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 19:08:33.539009 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.538986    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 19:08:33.539009 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.539008    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 19:08:33.539220 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.539017    2567 projected.go:196] Error preparing data for projected volume kube-api-access-wfmm2 for pod openshift-network-diagnostics/network-check-target-pksgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:33.539220 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:33.539049    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2 podName:cb4f382a-7ccc-47e1-bba5-b4ddb9956222 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:05.539042385 +0000 UTC m=+66.190668317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wfmm2" (UniqueName: "kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2") pod "network-check-target-pksgb" (UID: "cb4f382a-7ccc-47e1-bba5-b4ddb9956222") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 19:08:34.845114 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:34.844903    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:34.845624 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:34.844907    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:34.845624 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:34.845223    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:34.845624 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:34.845323    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:35.844886 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:35.844859    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:35.845044 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:35.844987    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:36.844759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:36.844730    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:36.845384 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:36.844729    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:36.845384 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:36.844855    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:36.845384 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:36.844901    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:37.844852 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:37.844814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:37.845343 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:37.844982    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cwklg" podUID="d9b59752-23f5-462c-a926-8716ec8155d2"
Jul 07 19:08:38.844995 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:38.844967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:38.845495 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:38.844969    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:38.845495 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:38.845069    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lg54c" podUID="86bce9f5-a068-47a9-bbdd-ef0784cf3415"
Jul 07 19:08:38.845495 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:38.845136    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pksgb" podUID="cb4f382a-7ccc-47e1-bba5-b4ddb9956222"
Jul 07 19:08:39.051810 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.051777    2567 generic.go:356] "Generic (PLEG): container finished" podID="615748c0-51f6-4d14-8991-4761f31ea68a" containerID="11694d23032295d5c39a3e9dc0fad1c0439304adf86788b44f291e9f375a208c" exitCode=0
Jul 07 19:08:39.051951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.051841    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerDied","Data":"11694d23032295d5c39a3e9dc0fad1c0439304adf86788b44f291e9f375a208c"}
Jul 07 19:08:39.128182 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.128160    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-187.ec2.internal" event="NodeReady"
Jul 07 19:08:39.128299 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.128289    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 19:08:39.182501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.182477    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7nsjg"]
Jul 07 19:08:39.202737 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.202713    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jg7vz"]
Jul 07 19:08:39.202834 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.202812    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.205033 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.205011    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 19:08:39.205139 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.205085    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 19:08:39.205139 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.205118    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42k7f\""
Jul 07 19:08:39.219355 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.219337    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7nsjg"]
Jul 07 19:08:39.219455 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.219365    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jg7vz"]
Jul 07 19:08:39.219455 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.219437    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.221610 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.221590    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 19:08:39.221708 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.221612    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jbcxp\""
Jul 07 19:08:39.221708 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.221642    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 19:08:39.221822 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.221738    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 19:08:39.384996 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.384901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46f326-2959-42bf-814d-bc4364f9a41d-tmp-dir\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.384996 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.384957    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b16b18-ef1e-49c5-ad84-f1e01935e6c6-cert\") pod \"ingress-canary-jg7vz\" (UID: \"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6\") " pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.385148 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.385006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26czs\" (UniqueName: \"kubernetes.io/projected/3a46f326-2959-42bf-814d-bc4364f9a41d-kube-api-access-26czs\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.385148 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.385037    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a46f326-2959-42bf-814d-bc4364f9a41d-config-volume\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.385148 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.385072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9rh\" (UniqueName: \"kubernetes.io/projected/a9b16b18-ef1e-49c5-ad84-f1e01935e6c6-kube-api-access-dw9rh\") pod \"ingress-canary-jg7vz\" (UID: \"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6\") " pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.385148 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.385103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.486160 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486134    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46f326-2959-42bf-814d-bc4364f9a41d-tmp-dir\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.486229 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b16b18-ef1e-49c5-ad84-f1e01935e6c6-cert\") pod \"ingress-canary-jg7vz\" (UID: \"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6\") " pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.486229 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486189    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26czs\" (UniqueName: \"kubernetes.io/projected/3a46f326-2959-42bf-814d-bc4364f9a41d-kube-api-access-26czs\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.486296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a46f326-2959-42bf-814d-bc4364f9a41d-config-volume\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.486296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9rh\" (UniqueName: \"kubernetes.io/projected/a9b16b18-ef1e-49c5-ad84-f1e01935e6c6-kube-api-access-dw9rh\") pod \"ingress-canary-jg7vz\" (UID: \"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6\") " pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.486372 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486309    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.486418 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.486385    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:08:39.486502 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.486486    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls podName:3a46f326-2959-42bf-814d-bc4364f9a41d nodeName:}" failed. No retries permitted until 2026-07-07 19:08:39.986429024 +0000 UTC m=+40.638054937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls") pod "dns-default-7nsjg" (UID: "3a46f326-2959-42bf-814d-bc4364f9a41d") : secret "dns-default-metrics-tls" not found
Jul 07 19:08:39.486640 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486513    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46f326-2959-42bf-814d-bc4364f9a41d-tmp-dir\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.486762 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.486745    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3a46f326-2959-42bf-814d-bc4364f9a41d-config-volume\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.490467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.490445    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9b16b18-ef1e-49c5-ad84-f1e01935e6c6-cert\") pod \"ingress-canary-jg7vz\" (UID: \"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6\") " pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.495342 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.495321    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9rh\" (UniqueName: \"kubernetes.io/projected/a9b16b18-ef1e-49c5-ad84-f1e01935e6c6-kube-api-access-dw9rh\") pod \"ingress-canary-jg7vz\" (UID: \"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6\") " pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.495661 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.495643    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26czs\" (UniqueName: \"kubernetes.io/projected/3a46f326-2959-42bf-814d-bc4364f9a41d-kube-api-access-26czs\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.506585 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.506561    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-648d497c84-bppp5"]
Jul 07 19:08:39.527884 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.527862    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jg7vz"
Jul 07 19:08:39.530670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.530653    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648d497c84-bppp5"]
Jul 07 19:08:39.530776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.530752    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.533211 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.533186    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 19:08:39.533369 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.533347    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-82tfj\""
Jul 07 19:08:39.533468 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.533428    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 19:08:39.533468 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.533436    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 19:08:39.542173 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.541598    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 19:08:39.688498 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-bound-sa-token\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.688670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688512    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f200738-f262-4fc8-9dd5-cf498dc3b77e-ca-trust-extracted\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.688670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55l6\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-kube-api-access-f55l6\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.688670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688576    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-trusted-ca\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.688670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-installation-pull-secrets\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.688810 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-image-registry-private-configuration\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.688843 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688810    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-certificates\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.689096 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.688846    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.707885 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.707865    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jg7vz"]
Jul 07 19:08:39.718906 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:39.718882    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b16b18_ef1e_49c5_ad84_f1e01935e6c6.slice/crio-03a269673f2dae71dcb3c11eed14ee4ce756750dacf3ada4fa95475ba97e29c8 WatchSource:0}: Error finding container 03a269673f2dae71dcb3c11eed14ee4ce756750dacf3ada4fa95475ba97e29c8: Status 404 returned error can't find the container with id 03a269673f2dae71dcb3c11eed14ee4ce756750dacf3ada4fa95475ba97e29c8
Jul 07 19:08:39.789084 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789055    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-trusted-ca\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789084 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789086    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-installation-pull-secrets\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789261 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789113    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-image-registry-private-configuration\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789345 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789323    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-certificates\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789444 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-bound-sa-token\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789495 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f200738-f262-4fc8-9dd5-cf498dc3b77e-ca-trust-extracted\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789495 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55l6\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-kube-api-access-f55l6\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.789590 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.789541    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 19:08:39.789590 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.789561    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d497c84-bppp5: secret "image-registry-tls" not found
Jul 07 19:08:39.789677 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.789629    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls podName:8f200738-f262-4fc8-9dd5-cf498dc3b77e nodeName:}" failed. No retries permitted until 2026-07-07 19:08:40.289607205 +0000 UTC m=+40.941233117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls") pod "image-registry-648d497c84-bppp5" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e") : secret "image-registry-tls" not found
Jul 07 19:08:39.790076 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789895    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-certificates\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.790076 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.789998    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-trusted-ca\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.790209 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.790098    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f200738-f262-4fc8-9dd5-cf498dc3b77e-ca-trust-extracted\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.792290 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.792268    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-image-registry-private-configuration\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.792557 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.792541    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-installation-pull-secrets\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.797882 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.797856    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-bound-sa-token\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.798238 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.798222    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55l6\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-kube-api-access-f55l6\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:39.845179 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.845157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:08:39.848203 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.848181    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 19:08:39.990178 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:39.990147    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:39.990312 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.990248    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:08:39.990312 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:39.990296    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls podName:3a46f326-2959-42bf-814d-bc4364f9a41d nodeName:}" failed. No retries permitted until 2026-07-07 19:08:40.990283342 +0000 UTC m=+41.641909252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls") pod "dns-default-7nsjg" (UID: "3a46f326-2959-42bf-814d-bc4364f9a41d") : secret "dns-default-metrics-tls" not found
Jul 07 19:08:40.055394 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.055365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jg7vz" event={"ID":"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6","Type":"ContainerStarted","Data":"03a269673f2dae71dcb3c11eed14ee4ce756750dacf3ada4fa95475ba97e29c8"}
Jul 07 19:08:40.058175 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.058153    2567 generic.go:356] "Generic (PLEG): container finished" podID="615748c0-51f6-4d14-8991-4761f31ea68a" containerID="7d2729a07d2aee911c8739373fb9c2b46c378763c8fcbb55660ebe2ad294fdda" exitCode=0
Jul 07 19:08:40.058269 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.058204    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerDied","Data":"7d2729a07d2aee911c8739373fb9c2b46c378763c8fcbb55660ebe2ad294fdda"}
Jul 07 19:08:40.292360 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.292292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:40.292497 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:40.292418    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 19:08:40.292497 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:40.292439    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d497c84-bppp5: secret "image-registry-tls" not found
Jul 07 19:08:40.292581 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:40.292500    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls podName:8f200738-f262-4fc8-9dd5-cf498dc3b77e nodeName:}" failed. No retries permitted until 2026-07-07 19:08:41.292482463 +0000 UTC m=+41.944108374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls") pod "image-registry-648d497c84-bppp5" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e") : secret "image-registry-tls" not found
Jul 07 19:08:40.844461 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.844424    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:08:40.844613 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.844471    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:08:40.847503 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.847474    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xjkd\""
Jul 07 19:08:40.847963 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.847516    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7q8zk\""
Jul 07 19:08:40.847963 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.847572    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 19:08:40.847963 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.847639    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 19:08:40.847963 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.847747    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 19:08:40.996626 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:40.996590    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:40.996760 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:40.996691    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:08:40.996760 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:40.996740    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls podName:3a46f326-2959-42bf-814d-bc4364f9a41d nodeName:}" failed. No retries permitted until 2026-07-07 19:08:42.996727278 +0000 UTC m=+43.648353189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls") pod "dns-default-7nsjg" (UID: "3a46f326-2959-42bf-814d-bc4364f9a41d") : secret "dns-default-metrics-tls" not found
Jul 07 19:08:41.066428 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:41.066397    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" event={"ID":"615748c0-51f6-4d14-8991-4761f31ea68a","Type":"ContainerStarted","Data":"46a5eb8a127b8c1fff734e333315c4fbdda3a2a8b8bd0ade7ebfa82bad2920be"}
Jul 07 19:08:41.093470 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:41.093421    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mxgpb" podStartSLOduration=9.036332962 podStartE2EDuration="41.093408033s" totalImagesPullingTime="32.057075071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:41.0932464 +0000 UTC m=+41.744872332" watchObservedRunningTime="2026-07-07 19:08:41.093408033 +0000 UTC m=+41.745033966"
Jul 07 19:08:41.298903 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:41.298871    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:41.299074 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:41.299022    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 19:08:41.299074 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:41.299040    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d497c84-bppp5: secret "image-registry-tls" not found
Jul 07 19:08:41.299161 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:41.299093    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls podName:8f200738-f262-4fc8-9dd5-cf498dc3b77e nodeName:}" failed. No retries permitted until 2026-07-07 19:08:43.299078153 +0000 UTC m=+43.950704068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls") pod "image-registry-648d497c84-bppp5" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e") : secret "image-registry-tls" not found
Jul 07 19:08:43.010860 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:43.010681    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:43.011252 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:43.010825    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:08:43.011252 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:43.010983    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls podName:3a46f326-2959-42bf-814d-bc4364f9a41d nodeName:}" failed. No retries permitted until 2026-07-07 19:08:47.010967951 +0000 UTC m=+47.662593863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls") pod "dns-default-7nsjg" (UID: "3a46f326-2959-42bf-814d-bc4364f9a41d") : secret "dns-default-metrics-tls" not found
Jul 07 19:08:43.072939 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:43.072898    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jg7vz" event={"ID":"a9b16b18-ef1e-49c5-ad84-f1e01935e6c6","Type":"ContainerStarted","Data":"b19692add477c8c6060a65c4b20891efe9c5da915bf998a52a38252970afdd95"}
Jul 07 19:08:43.091904 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:43.091858    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jg7vz" podStartSLOduration=1.421560058 podStartE2EDuration="4.091843648s" totalImagesPullingTime="2.67028359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:43.091306448 +0000 UTC m=+43.742932384" watchObservedRunningTime="2026-07-07 19:08:43.091843648 +0000 UTC m=+43.743469580"
Jul 07 19:08:43.313470 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:43.313430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:43.313666 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:43.313554    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 19:08:43.313666 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:43.313566    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d497c84-bppp5: secret "image-registry-tls" not found
Jul 07 19:08:43.313666 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:43.313611    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls podName:8f200738-f262-4fc8-9dd5-cf498dc3b77e nodeName:}" failed. No retries permitted until 2026-07-07 19:08:47.313599294 +0000 UTC m=+47.965225205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls") pod "image-registry-648d497c84-bppp5" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e") : secret "image-registry-tls" not found
Jul 07 19:08:47.037452 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:47.037416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:47.037976 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:47.037585    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:08:47.037976 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:47.037656    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls podName:3a46f326-2959-42bf-814d-bc4364f9a41d nodeName:}" failed. No retries permitted until 2026-07-07 19:08:55.03763926 +0000 UTC m=+55.689265193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls") pod "dns-default-7nsjg" (UID: "3a46f326-2959-42bf-814d-bc4364f9a41d") : secret "dns-default-metrics-tls" not found
Jul 07 19:08:47.339498 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:47.339463    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:47.339637 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:47.339620    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 19:08:47.339680 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:47.339639    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d497c84-bppp5: secret "image-registry-tls" not found
Jul 07 19:08:47.339732 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:47.339703    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls podName:8f200738-f262-4fc8-9dd5-cf498dc3b77e nodeName:}" failed. No retries permitted until 2026-07-07 19:08:55.33968545 +0000 UTC m=+55.991311361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls") pod "image-registry-648d497c84-bppp5" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e") : secret "image-registry-tls" not found
Jul 07 19:08:48.482980 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.482943    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"]
Jul 07 19:08:48.515017 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.514980    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"]
Jul 07 19:08:48.515146 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.515093    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"
Jul 07 19:08:48.517447 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.517409    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 07 19:08:48.517447 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.517417    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 07 19:08:48.518327 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.518308    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nmwzn\""
Jul 07 19:08:48.548025 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.547998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29tc\" (UniqueName: \"kubernetes.io/projected/29bb76d8-a742-4bb5-a4ae-76a124c9aa59-kube-api-access-t29tc\") pod \"migrator-d57d58cc9-qwl66\" (UID: \"29bb76d8-a742-4bb5-a4ae-76a124c9aa59\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"
Jul 07 19:08:48.649097 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.649071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29tc\" (UniqueName: \"kubernetes.io/projected/29bb76d8-a742-4bb5-a4ae-76a124c9aa59-kube-api-access-t29tc\") pod \"migrator-d57d58cc9-qwl66\" (UID: \"29bb76d8-a742-4bb5-a4ae-76a124c9aa59\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"
Jul 07 19:08:48.661523 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.661498    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29tc\" (UniqueName: \"kubernetes.io/projected/29bb76d8-a742-4bb5-a4ae-76a124c9aa59-kube-api-access-t29tc\") pod \"migrator-d57d58cc9-qwl66\" (UID: \"29bb76d8-a742-4bb5-a4ae-76a124c9aa59\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"
Jul 07 19:08:48.683208 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.683184    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nwrdf"]
Jul 07 19:08:48.700271 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.700252    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.703617 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.703589    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwrdf"]
Jul 07 19:08:48.703901 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.703883    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 19:08:48.703997 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.703935    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kjmsg\""
Jul 07 19:08:48.704061 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.703934    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 19:08:48.704221 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.704205    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 19:08:48.704280 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.704239    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 19:08:48.749587 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.749532    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-crio-socket\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.749587 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.749573    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.749761 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.749590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlvlk\" (UniqueName: \"kubernetes.io/projected/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-api-access-vlvlk\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.749761 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.749671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.749761 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.749754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-data-volume\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.824402 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.824377    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"
Jul 07 19:08:48.851039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.851108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851053    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlvlk\" (UniqueName: \"kubernetes.io/projected/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-api-access-vlvlk\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.851187 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:48.851169    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:48.851250 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851229    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.851307 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:48.851245    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:49.351224114 +0000 UTC m=+50.002850040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:48.851373 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:48.851349    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:08:48.851373 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851364    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-data-volume\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.851579 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-crio-socket\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.851579 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:48.851428    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:49.351410046 +0000 UTC m=+50.003035979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : configmap "kube-rbac-proxy" not found
Jul 07 19:08:48.851674 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851634    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-crio-socket\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.851712 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.851691    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-data-volume\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.861971 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.861950    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlvlk\" (UniqueName: \"kubernetes.io/projected/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-api-access-vlvlk\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:48.966626 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:48.966595    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66"]
Jul 07 19:08:48.969277 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:48.969252    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bb76d8_a742_4bb5_a4ae_76a124c9aa59.slice/crio-05e0d14feae642e8b56cc749147c0e4a5549793ad68cea7f292302d31050e3fa WatchSource:0}: Error finding container 05e0d14feae642e8b56cc749147c0e4a5549793ad68cea7f292302d31050e3fa: Status 404 returned error can't find the container with id 05e0d14feae642e8b56cc749147c0e4a5549793ad68cea7f292302d31050e3fa
Jul 07 19:08:49.085849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:49.085779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66" event={"ID":"29bb76d8-a742-4bb5-a4ae-76a124c9aa59","Type":"ContainerStarted","Data":"05e0d14feae642e8b56cc749147c0e4a5549793ad68cea7f292302d31050e3fa"}
Jul 07 19:08:49.354849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:49.354766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:49.354849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:49.354822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:49.355090 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:49.354933    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:08:49.355090 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:49.354960    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:49.355090 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:49.354987    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:50.354972731 +0000 UTC m=+51.006598641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : configmap "kube-rbac-proxy" not found
Jul 07 19:08:49.355090 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:49.355024    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:50.355004542 +0000 UTC m=+51.006630453 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:50.363520 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.363478    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:50.364117 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.363572    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:50.364117 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:50.363605    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:08:50.364117 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:50.363673    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:50.364117 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:50.363702    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:52.363679036 +0000 UTC m=+53.015304949 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : configmap "kube-rbac-proxy" not found
Jul 07 19:08:50.364117 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:50.363723    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:52.363713745 +0000 UTC m=+53.015339659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:50.865214 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.865183    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6kfr4_4acae846-0bb0-41fa-af29-3ec6eeb83eee/dns-node-resolver/0.log"
Jul 07 19:08:50.961094 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.961065    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fft46"]
Jul 07 19:08:50.975142 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.975118    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fft46"]
Jul 07 19:08:50.975275 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.975219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:50.977876 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.977853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 19:08:50.978002 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.977856    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 19:08:50.978002 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.977856    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vblph\""
Jul 07 19:08:50.978711 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.978694    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 19:08:50.978757 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:50.978741    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 19:08:51.069315 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.069287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-signing-cabundle\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.069315 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.069316    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-signing-key\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.069495 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.069344    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bjv\" (UniqueName: \"kubernetes.io/projected/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-kube-api-access-c9bjv\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.170724 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.170698    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-signing-cabundle\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.170857 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.170739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-signing-key\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.170857 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.170794    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bjv\" (UniqueName: \"kubernetes.io/projected/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-kube-api-access-c9bjv\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.171417 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.171396    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-signing-cabundle\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.173157 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.173138    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-signing-key\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.178223 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.178200    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bjv\" (UniqueName: \"kubernetes.io/projected/55b95d3b-bf09-4eee-a02c-ebf9ce052dd2-kube-api-access-c9bjv\") pod \"service-ca-758f599c5-fft46\" (UID: \"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2\") " pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.284419 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.284389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-fft46"
Jul 07 19:08:51.413102 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.413023    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fft46"]
Jul 07 19:08:51.417419 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:08:51.417391    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b95d3b_bf09_4eee_a02c_ebf9ce052dd2.slice/crio-2468ad6dc06bf0c63bbfdf40e418c65a842788292f19bb19fbb37f02877cf63c WatchSource:0}: Error finding container 2468ad6dc06bf0c63bbfdf40e418c65a842788292f19bb19fbb37f02877cf63c: Status 404 returned error can't find the container with id 2468ad6dc06bf0c63bbfdf40e418c65a842788292f19bb19fbb37f02877cf63c
Jul 07 19:08:51.864666 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:51.864638    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5hw59_8700ea39-999a-468b-b53e-5cabbcc4b5ed/node-ca/0.log"
Jul 07 19:08:52.095242 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.095178    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-fft46" event={"ID":"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2","Type":"ContainerStarted","Data":"2468ad6dc06bf0c63bbfdf40e418c65a842788292f19bb19fbb37f02877cf63c"}
Jul 07 19:08:52.097015 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.096990    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66" event={"ID":"29bb76d8-a742-4bb5-a4ae-76a124c9aa59","Type":"ContainerStarted","Data":"69cd30f2067ba49e0aa0ed7ed2bfdee6783ca7f66eb19049a2b4cfa473907963"}
Jul 07 19:08:52.097136 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.097035    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66" event={"ID":"29bb76d8-a742-4bb5-a4ae-76a124c9aa59","Type":"ContainerStarted","Data":"2080b32d4dc41266acdcc36fe232431cd035b8840042df0d3ff19cea3bf10cd8"}
Jul 07 19:08:52.114256 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.114204    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qwl66" podStartSLOduration=2.000789437 podStartE2EDuration="4.114184897s" totalImagesPullingTime="2.11339546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:52.113467323 +0000 UTC m=+52.765093256" watchObservedRunningTime="2026-07-07 19:08:52.114184897 +0000 UTC m=+52.765810832"
Jul 07 19:08:52.381729 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.381692    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:52.381890 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.381747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:52.381890 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:52.381816    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:08:52.381890 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:52.381841    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:52.381890 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:52.381877    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:56.381862063 +0000 UTC m=+57.033487979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : configmap "kube-rbac-proxy" not found
Jul 07 19:08:52.381890 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:52.381892    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:08:56.381886104 +0000 UTC m=+57.033512015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:52.665112 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:52.665033    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jg7vz_a9b16b18-ef1e-49c5-ad84-f1e01935e6c6/serve-healthcheck-canary/0.log"
Jul 07 19:08:53.065733 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:53.065707    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-qwl66_29bb76d8-a742-4bb5-a4ae-76a124c9aa59/migrator/0.log"
Jul 07 19:08:53.268598 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:53.268514    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-qwl66_29bb76d8-a742-4bb5-a4ae-76a124c9aa59/graceful-termination/0.log"
Jul 07 19:08:54.103627 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:54.103587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-fft46" event={"ID":"55b95d3b-bf09-4eee-a02c-ebf9ce052dd2","Type":"ContainerStarted","Data":"4f6dc36805657aa4fe7367e7fc292936aedb8fff5d51142c3e82916cf0efc5f1"}
Jul 07 19:08:55.101801 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:55.101764    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:08:55.102000 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:55.101951    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 19:08:55.102059 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:55.102017    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls podName:3a46f326-2959-42bf-814d-bc4364f9a41d nodeName:}" failed. No retries permitted until 2026-07-07 19:09:11.102000508 +0000 UTC m=+71.753626418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls") pod "dns-default-7nsjg" (UID: "3a46f326-2959-42bf-814d-bc4364f9a41d") : secret "dns-default-metrics-tls" not found
Jul 07 19:08:55.404852 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:55.404750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:08:55.405311 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:55.404936    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 07 19:08:55.405311 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:55.404953    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-648d497c84-bppp5: secret "image-registry-tls" not found
Jul 07 19:08:55.405311 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:55.405007    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls podName:8f200738-f262-4fc8-9dd5-cf498dc3b77e nodeName:}" failed. No retries permitted until 2026-07-07 19:09:11.404990169 +0000 UTC m=+72.056616083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls") pod "image-registry-648d497c84-bppp5" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e") : secret "image-registry-tls" not found
Jul 07 19:08:56.413075 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:56.413038    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:56.413444 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:08:56.413117    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:08:56.413444 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:56.413184    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 19:08:56.413444 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:56.413213    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:08:56.413444 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:56.413264    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:04.413244208 +0000 UTC m=+65.064870122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : configmap "kube-rbac-proxy" not found
Jul 07 19:08:56.413444 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:08:56.413283    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:04.413273912 +0000 UTC m=+65.064899833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:09:04.483734 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:04.483693    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:09:04.484315 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:04.483756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:09:04.484315 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:09:04.483844    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 19:09:04.484315 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:09:04.483909    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls podName:3cbbecc2-3a6b-4e03-9060-efc6f1675ae2 nodeName:}" failed. No retries permitted until 2026-07-07 19:09:20.483893584 +0000 UTC m=+81.135519498 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nwrdf" (UID: "3cbbecc2-3a6b-4e03-9060-efc6f1675ae2") : secret "insights-runtime-extractor-tls" not found
Jul 07 19:09:04.484315 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:04.484227    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:09:05.052214 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.052184    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47hd8"
Jul 07 19:09:05.083152 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.083105    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-fft46" podStartSLOduration=12.796782669 podStartE2EDuration="15.083091055s" totalImagesPullingTime="2.286308386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:54.119186957 +0000 UTC m=+54.770812911" watchObservedRunningTime="2026-07-07 19:09:05.083091055 +0000 UTC m=+65.734717051"
Jul 07 19:09:05.591698 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.591662    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:09:05.592177 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.591718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:09:05.592177 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.591820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:09:05.594584 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.594566    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 19:09:05.594630 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.594571    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 19:09:05.594662 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.594632    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 19:09:05.604849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.604819    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d9b59752-23f5-462c-a926-8716ec8155d2-original-pull-secret\") pod \"global-pull-secret-syncer-cwklg\" (UID: \"d9b59752-23f5-462c-a926-8716ec8155d2\") " pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:09:05.604999 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.604908    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86bce9f5-a068-47a9-bbdd-ef0784cf3415-metrics-certs\") pod \"network-metrics-daemon-lg54c\" (UID: \"86bce9f5-a068-47a9-bbdd-ef0784cf3415\") " pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:09:05.605086 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.605070    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 19:09:05.615381 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.615361    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmm2\" (UniqueName: \"kubernetes.io/projected/cb4f382a-7ccc-47e1-bba5-b4ddb9956222-kube-api-access-wfmm2\") pod \"network-check-target-pksgb\" (UID: \"cb4f382a-7ccc-47e1-bba5-b4ddb9956222\") " pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:09:05.654757 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.654732    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cwklg"
Jul 07 19:09:05.759158 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.759131    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xjkd\""
Jul 07 19:09:05.765833 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.765809    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7q8zk\""
Jul 07 19:09:05.766824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.766809    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lg54c"
Jul 07 19:09:05.774228 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.774172    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:09:05.775283 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.775262    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cwklg"]
Jul 07 19:09:05.778253 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:05.778228    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b59752_23f5_462c_a926_8716ec8155d2.slice/crio-1ab6962a14dd3aec61a187e64ad42a37ff9a6f243a042ca149a8627115a34e8b WatchSource:0}: Error finding container 1ab6962a14dd3aec61a187e64ad42a37ff9a6f243a042ca149a8627115a34e8b: Status 404 returned error can't find the container with id 1ab6962a14dd3aec61a187e64ad42a37ff9a6f243a042ca149a8627115a34e8b
Jul 07 19:09:05.898895 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.898861    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lg54c"]
Jul 07 19:09:05.902547 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:05.902523    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bce9f5_a068_47a9_bbdd_ef0784cf3415.slice/crio-ccc61e5d0e03cf7fa403838ba0b6b0277bbe5ea206c31b1ee34b6e344be79208 WatchSource:0}: Error finding container ccc61e5d0e03cf7fa403838ba0b6b0277bbe5ea206c31b1ee34b6e344be79208: Status 404 returned error can't find the container with id ccc61e5d0e03cf7fa403838ba0b6b0277bbe5ea206c31b1ee34b6e344be79208
Jul 07 19:09:05.915123 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:05.915100    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pksgb"]
Jul 07 19:09:05.918157 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:05.918135    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4f382a_7ccc_47e1_bba5_b4ddb9956222.slice/crio-8169f62217e2577bfe79075733c24ecd36e49b1ecf9cba4679a5f58f6542175d WatchSource:0}: Error finding container 8169f62217e2577bfe79075733c24ecd36e49b1ecf9cba4679a5f58f6542175d: Status 404 returned error can't find the container with id 8169f62217e2577bfe79075733c24ecd36e49b1ecf9cba4679a5f58f6542175d
Jul 07 19:09:06.134514 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:06.134426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pksgb" event={"ID":"cb4f382a-7ccc-47e1-bba5-b4ddb9956222","Type":"ContainerStarted","Data":"8169f62217e2577bfe79075733c24ecd36e49b1ecf9cba4679a5f58f6542175d"}
Jul 07 19:09:06.135547 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:06.135519    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lg54c" event={"ID":"86bce9f5-a068-47a9-bbdd-ef0784cf3415","Type":"ContainerStarted","Data":"ccc61e5d0e03cf7fa403838ba0b6b0277bbe5ea206c31b1ee34b6e344be79208"}
Jul 07 19:09:06.136492 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:06.136471    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cwklg" event={"ID":"d9b59752-23f5-462c-a926-8716ec8155d2","Type":"ContainerStarted","Data":"1ab6962a14dd3aec61a187e64ad42a37ff9a6f243a042ca149a8627115a34e8b"}
Jul 07 19:09:08.146425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:08.146365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lg54c" event={"ID":"86bce9f5-a068-47a9-bbdd-ef0784cf3415","Type":"ContainerStarted","Data":"6ddd6c3091663f4268c117ec03346cd08df26129df352b3ffc0e72a2568dd11b"}
Jul 07 19:09:08.146891 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:08.146450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lg54c" event={"ID":"86bce9f5-a068-47a9-bbdd-ef0784cf3415","Type":"ContainerStarted","Data":"90fc5727fe72e148e79d666707a3358318fc19638c432a06526631597390e1f0"}
Jul 07 19:09:08.164552 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:08.164493    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lg54c" podStartSLOduration=66.786327172 podStartE2EDuration="1m8.164474681s" totalImagesPullingTime="1.378147509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:08.164196682 +0000 UTC m=+68.815822616" watchObservedRunningTime="2026-07-07 19:09:08.164474681 +0000 UTC m=+68.816100615"
Jul 07 19:09:10.154983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:10.154943    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pksgb" event={"ID":"cb4f382a-7ccc-47e1-bba5-b4ddb9956222","Type":"ContainerStarted","Data":"d59c9d637c2db76f1540d6c2cfb90575f996b245eab7c95bdaf96bb33e6d1a73"}
Jul 07 19:09:10.155413 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:10.155088    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:09:10.174973 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:10.174904    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pksgb" podStartSLOduration=66.388207685 podStartE2EDuration="1m10.174887019s" totalImagesPullingTime="3.786679334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:10.174719552 +0000 UTC m=+70.826345489" watchObservedRunningTime="2026-07-07 19:09:10.174887019 +0000 UTC m=+70.826512953"
Jul 07 19:09:11.137991 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.137902    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:09:11.140043 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.140014    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3a46f326-2959-42bf-814d-bc4364f9a41d-metrics-tls\") pod \"dns-default-7nsjg\" (UID: \"3a46f326-2959-42bf-814d-bc4364f9a41d\") " pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:09:11.313824 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.313798    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-42k7f\""
Jul 07 19:09:11.321951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.321929    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:09:11.440232 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.440205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:11.442471 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.442445    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"image-registry-648d497c84-bppp5\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") " pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:11.452607 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.452532    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7nsjg"]
Jul 07 19:09:11.455656 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:11.455630    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a46f326_2959_42bf_814d_bc4364f9a41d.slice/crio-669e86a91af8ef9fd8ce79c760026f83ec8426ce887e01c7e05543514ed536c5 WatchSource:0}: Error finding container 669e86a91af8ef9fd8ce79c760026f83ec8426ce887e01c7e05543514ed536c5: Status 404 returned error can't find the container with id 669e86a91af8ef9fd8ce79c760026f83ec8426ce887e01c7e05543514ed536c5
Jul 07 19:09:11.642994 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.642965    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-82tfj\""
Jul 07 19:09:11.651182 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.651161    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:11.774880 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:11.774850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-648d497c84-bppp5"]
Jul 07 19:09:11.777674 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:11.777651    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f200738_f262_4fc8_9dd5_cf498dc3b77e.slice/crio-e0373a686372f4af50a81783f663c6d3521d3b39d24321d70d5d493be81f2c88 WatchSource:0}: Error finding container e0373a686372f4af50a81783f663c6d3521d3b39d24321d70d5d493be81f2c88: Status 404 returned error can't find the container with id e0373a686372f4af50a81783f663c6d3521d3b39d24321d70d5d493be81f2c88
Jul 07 19:09:12.163615 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:12.163576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d497c84-bppp5" event={"ID":"8f200738-f262-4fc8-9dd5-cf498dc3b77e","Type":"ContainerStarted","Data":"f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e"}
Jul 07 19:09:12.163793 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:12.163636    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d497c84-bppp5" event={"ID":"8f200738-f262-4fc8-9dd5-cf498dc3b77e","Type":"ContainerStarted","Data":"e0373a686372f4af50a81783f663c6d3521d3b39d24321d70d5d493be81f2c88"}
Jul 07 19:09:12.163793 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:12.163681    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:12.164717 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:12.164696    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nsjg" event={"ID":"3a46f326-2959-42bf-814d-bc4364f9a41d","Type":"ContainerStarted","Data":"669e86a91af8ef9fd8ce79c760026f83ec8426ce887e01c7e05543514ed536c5"}
Jul 07 19:09:12.166140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:12.166122    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cwklg" event={"ID":"d9b59752-23f5-462c-a926-8716ec8155d2","Type":"ContainerStarted","Data":"8297131c77c359e2dc0660d0315a6334a375d5b6526a845d6f9feb4cae057747"}
Jul 07 19:09:12.228053 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:12.227967    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-648d497c84-bppp5" podStartSLOduration=33.22794572 podStartE2EDuration="33.22794572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:12.20085309 +0000 UTC m=+72.852479024" watchObservedRunningTime="2026-07-07 19:09:12.22794572 +0000 UTC m=+72.879571653"
Jul 07 19:09:14.176770 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:14.176740    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nsjg" event={"ID":"3a46f326-2959-42bf-814d-bc4364f9a41d","Type":"ContainerStarted","Data":"6cf86b445e2c228f2a4eebf134ac57417efb5dd46f57ec5e8c79d3903ada91ff"}
Jul 07 19:09:14.177181 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:14.176794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7nsjg" event={"ID":"3a46f326-2959-42bf-814d-bc4364f9a41d","Type":"ContainerStarted","Data":"b8723f5afbe5d4f862f757498f29018f925dcc7e0268f9c8901ab3a86dfd7f19"}
Jul 07 19:09:14.177181 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:14.176877    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:09:14.197676 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:14.197633    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cwklg" podStartSLOduration=68.787445093 podStartE2EDuration="1m14.197619192s" totalImagesPullingTime="5.410174099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:12.227217135 +0000 UTC m=+72.878843069" watchObservedRunningTime="2026-07-07 19:09:14.197619192 +0000 UTC m=+74.849245125"
Jul 07 19:09:14.197859 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:14.197840    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7nsjg" podStartSLOduration=32.771060435 podStartE2EDuration="35.19783385s" totalImagesPullingTime="2.426773415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:14.19704405 +0000 UTC m=+74.848669983" watchObservedRunningTime="2026-07-07 19:09:14.19783385 +0000 UTC m=+74.849459782"
Jul 07 19:09:14.987057 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:14.987014    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"]
Jul 07 19:09:15.008802 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.008765    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"]
Jul 07 19:09:15.008972 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.008888    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.011667 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.011642    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 19:09:15.011667 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.011643    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 19:09:15.012192 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.012170    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 19:09:15.012310 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.012178    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 19:09:15.023899 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.023879    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"]
Jul 07 19:09:15.024007 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.023937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"]
Jul 07 19:09:15.024007 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.023998    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.026306 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.026288    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 07 19:09:15.026306 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.026297    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 07 19:09:15.026447 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.026335    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 07 19:09:15.026447 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.026337    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 07 19:09:15.168671 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168635    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648d497c84-bppp5"]
Jul 07 19:09:15.168958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168711    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.168958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168738    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-hub\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.168958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168759    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.168958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.168958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168810    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ceb506f-dcb7-4694-9c86-7f9746fee448-tmp\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.168958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168837    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-ca\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.169226 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.168992    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5ceb506f-dcb7-4694-9c86-7f9746fee448-klusterlet-config\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.169226 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.169044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwslv\" (UniqueName: \"kubernetes.io/projected/5ceb506f-dcb7-4694-9c86-7f9746fee448-kube-api-access-vwslv\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.169226 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.169075    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfrf\" (UniqueName: \"kubernetes.io/projected/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-kube-api-access-qhfrf\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.269530 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269456    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b87dd9f8b-zxwpl"]
Jul 07 19:09:15.269530 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269501    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-hub\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ceb506f-dcb7-4694-9c86-7f9746fee448-tmp\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-ca\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5ceb506f-dcb7-4694-9c86-7f9746fee448-klusterlet-config\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269698    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwslv\" (UniqueName: \"kubernetes.io/projected/5ceb506f-dcb7-4694-9c86-7f9746fee448-kube-api-access-vwslv\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfrf\" (UniqueName: \"kubernetes.io/projected/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-kube-api-access-qhfrf\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.269774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.270039 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.270032    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ceb506f-dcb7-4694-9c86-7f9746fee448-tmp\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.270548 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.270364    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.272588 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.272563    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5ceb506f-dcb7-4694-9c86-7f9746fee448-klusterlet-config\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.272588 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.272580    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.272778 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.272732    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.272778 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.272737    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-ca\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.272778 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.272760    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-hub\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.273384 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.273369    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.282742 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.282720    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwslv\" (UniqueName: \"kubernetes.io/projected/5ceb506f-dcb7-4694-9c86-7f9746fee448-kube-api-access-vwslv\") pod \"klusterlet-addon-workmgr-8655cf486c-z58k8\" (UID: \"5ceb506f-dcb7-4694-9c86-7f9746fee448\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.283671 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.283653    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfrf\" (UniqueName: \"kubernetes.io/projected/aa68162d-ca1c-4c68-8c6a-64812b7a1d66-kube-api-access-qhfrf\") pod \"cluster-proxy-proxy-agent-579455cdf9-5cgjs\" (UID: \"aa68162d-ca1c-4c68-8c6a-64812b7a1d66\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.286658 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.286635    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b87dd9f8b-zxwpl"]
Jul 07 19:09:15.320018 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.319991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:15.340149 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.340118    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"
Jul 07 19:09:15.371165 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371130    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ba6eff-2549-4380-9f07-9e2ee46720b0-ca-trust-extracted\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ba6eff-2549-4380-9f07-9e2ee46720b0-image-registry-private-configuration\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-registry-tls\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371264    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ba6eff-2549-4380-9f07-9e2ee46720b0-registry-certificates\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371293    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjxv\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-kube-api-access-ffjxv\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ba6eff-2549-4380-9f07-9e2ee46720b0-trusted-ca\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-bound-sa-token\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.371501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.371417    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ba6eff-2549-4380-9f07-9e2ee46720b0-installation-pull-secrets\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.450822 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.450792    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"]
Jul 07 19:09:15.453171 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:15.453138    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceb506f_dcb7_4694_9c86_7f9746fee448.slice/crio-894c8a8fbc1e5fe23b873bc850d77c07cca9fd5bebcd9f730f228fe85b711d2c WatchSource:0}: Error finding container 894c8a8fbc1e5fe23b873bc850d77c07cca9fd5bebcd9f730f228fe85b711d2c: Status 404 returned error can't find the container with id 894c8a8fbc1e5fe23b873bc850d77c07cca9fd5bebcd9f730f228fe85b711d2c
Jul 07 19:09:15.469961 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.469937    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs"]
Jul 07 19:09:15.471931 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.471894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ba6eff-2549-4380-9f07-9e2ee46720b0-ca-trust-extracted\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.471985 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.471966    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ba6eff-2549-4380-9f07-9e2ee46720b0-image-registry-private-configuration\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472021 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-registry-tls\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472066 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ba6eff-2549-4380-9f07-9e2ee46720b0-registry-certificates\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472103 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472091    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjxv\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-kube-api-access-ffjxv\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472145 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472132    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ba6eff-2549-4380-9f07-9e2ee46720b0-trusted-ca\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472180 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472164    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-bound-sa-token\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472259 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472222    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ba6eff-2549-4380-9f07-9e2ee46720b0-installation-pull-secrets\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.472351 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.472275    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ba6eff-2549-4380-9f07-9e2ee46720b0-ca-trust-extracted\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.473190 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.473167    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ba6eff-2549-4380-9f07-9e2ee46720b0-registry-certificates\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.473538 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.473506    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ba6eff-2549-4380-9f07-9e2ee46720b0-trusted-ca\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.473850 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:15.473828    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa68162d_ca1c_4c68_8c6a_64812b7a1d66.slice/crio-19069f978d37654bc5e8cdbe65e8f95aa50685eefd1f7b4a27cc840e97d3f4e9 WatchSource:0}: Error finding container 19069f978d37654bc5e8cdbe65e8f95aa50685eefd1f7b4a27cc840e97d3f4e9: Status 404 returned error can't find the container with id 19069f978d37654bc5e8cdbe65e8f95aa50685eefd1f7b4a27cc840e97d3f4e9
Jul 07 19:09:15.474593 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.474568    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ba6eff-2549-4380-9f07-9e2ee46720b0-installation-pull-secrets\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.474593 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.474586    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ba6eff-2549-4380-9f07-9e2ee46720b0-image-registry-private-configuration\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.474714 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.474620    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-registry-tls\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.479186 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.479165    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-bound-sa-token\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.479896 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.479878    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjxv\" (UniqueName: \"kubernetes.io/projected/09ba6eff-2549-4380-9f07-9e2ee46720b0-kube-api-access-ffjxv\") pod \"image-registry-b87dd9f8b-zxwpl\" (UID: \"09ba6eff-2549-4380-9f07-9e2ee46720b0\") " pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.598964 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.598880    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:15.743826 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:15.743795    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b87dd9f8b-zxwpl"]
Jul 07 19:09:15.746975 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:15.746947    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ba6eff_2549_4380_9f07_9e2ee46720b0.slice/crio-a4eed173a46e6cb412887daf203d066e94df66fccd9719f4acef7571b468e345 WatchSource:0}: Error finding container a4eed173a46e6cb412887daf203d066e94df66fccd9719f4acef7571b468e345: Status 404 returned error can't find the container with id a4eed173a46e6cb412887daf203d066e94df66fccd9719f4acef7571b468e345
Jul 07 19:09:16.185382 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:16.185341    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs" event={"ID":"aa68162d-ca1c-4c68-8c6a-64812b7a1d66","Type":"ContainerStarted","Data":"19069f978d37654bc5e8cdbe65e8f95aa50685eefd1f7b4a27cc840e97d3f4e9"}
Jul 07 19:09:16.186779 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:16.186737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl" event={"ID":"09ba6eff-2549-4380-9f07-9e2ee46720b0","Type":"ContainerStarted","Data":"ada43ea93a650ca181426f69d97e83f0d6a2b0c4cb8bf637a435551490db6ed6"}
Jul 07 19:09:16.186929 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:16.186801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl" event={"ID":"09ba6eff-2549-4380-9f07-9e2ee46720b0","Type":"ContainerStarted","Data":"a4eed173a46e6cb412887daf203d066e94df66fccd9719f4acef7571b468e345"}
Jul 07 19:09:16.186929 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:16.186870    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:16.187899 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:16.187872    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8" event={"ID":"5ceb506f-dcb7-4694-9c86-7f9746fee448","Type":"ContainerStarted","Data":"894c8a8fbc1e5fe23b873bc850d77c07cca9fd5bebcd9f730f228fe85b711d2c"}
Jul 07 19:09:16.223415 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:16.223358    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl" podStartSLOduration=1.223341295 podStartE2EDuration="1.223341295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:16.220162861 +0000 UTC m=+76.871788795" watchObservedRunningTime="2026-07-07 19:09:16.223341295 +0000 UTC m=+76.874967225"
Jul 07 19:09:20.518515 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:20.518473    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:09:20.521127 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:20.521098    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cbbecc2-3a6b-4e03-9060-efc6f1675ae2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nwrdf\" (UID: \"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2\") " pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:09:20.811967 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:20.811944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kjmsg\""
Jul 07 19:09:20.820219 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:20.820196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nwrdf"
Jul 07 19:09:20.977188 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:20.977159    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nwrdf"]
Jul 07 19:09:20.980875 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:20.980853    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbbecc2_3a6b_4e03_9060_efc6f1675ae2.slice/crio-fa8194e933d02837081fabde3059e60a2c0ef9534734f164caa2337f4a37b8e9 WatchSource:0}: Error finding container fa8194e933d02837081fabde3059e60a2c0ef9534734f164caa2337f4a37b8e9: Status 404 returned error can't find the container with id fa8194e933d02837081fabde3059e60a2c0ef9534734f164caa2337f4a37b8e9
Jul 07 19:09:21.210309 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.210222    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs" event={"ID":"aa68162d-ca1c-4c68-8c6a-64812b7a1d66","Type":"ContainerStarted","Data":"75a7e4805b49a9b66951d39333753aa3b9917db2991463eab54ecda881ad67dd"}
Jul 07 19:09:21.210309 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.210269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs" event={"ID":"aa68162d-ca1c-4c68-8c6a-64812b7a1d66","Type":"ContainerStarted","Data":"1666652fdf7711621192a7e2402641ee8e83e7a8e8fa7270b347017dc3f0d071"}
Jul 07 19:09:21.210309 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.210282    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs" event={"ID":"aa68162d-ca1c-4c68-8c6a-64812b7a1d66","Type":"ContainerStarted","Data":"f74d1dc603ff0d7f2d43354f3373f16788520ffe065dd785b99740d77e359940"}
Jul 07 19:09:21.211322 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.211293    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwrdf" event={"ID":"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2","Type":"ContainerStarted","Data":"fa8194e933d02837081fabde3059e60a2c0ef9534734f164caa2337f4a37b8e9"}
Jul 07 19:09:21.212585 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.212562    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8" event={"ID":"5ceb506f-dcb7-4694-9c86-7f9746fee448","Type":"ContainerStarted","Data":"65e2943e11fcf4199a6c88b771e6133c117bdc3df1347dbd9985f53d1493d4de"}
Jul 07 19:09:21.212775 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.212757    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:21.214185 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.214168    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8"
Jul 07 19:09:21.230018 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.229977    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579455cdf9-5cgjs" podStartSLOduration=1.96517454 podStartE2EDuration="7.229965008s" totalImagesPullingTime="5.264790468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:21.22984092 +0000 UTC m=+81.881466855" watchObservedRunningTime="2026-07-07 19:09:21.229965008 +0000 UTC m=+81.881590941"
Jul 07 19:09:21.251350 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:21.251305    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8655cf486c-z58k8" podStartSLOduration=1.968589892 podStartE2EDuration="7.25128883s" totalImagesPullingTime="5.282698938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:21.250717254 +0000 UTC m=+81.902343217" watchObservedRunningTime="2026-07-07 19:09:21.25128883 +0000 UTC m=+81.902914763"
Jul 07 19:09:24.182911 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:24.182782    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7nsjg"
Jul 07 19:09:25.231232 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:25.231204    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwrdf" event={"ID":"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2","Type":"ContainerStarted","Data":"d9c3e9b9a02b567bd28e21ae7b3928a6cbc9508edb9116dc2651d78d8a7910fe"}
Jul 07 19:09:26.237095 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:26.237060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwrdf" event={"ID":"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2","Type":"ContainerStarted","Data":"3ee04c04f2105276013d5eb459c0979bcf22c6bce3d1c64b244ccfaab12d6419"}
Jul 07 19:09:28.246367 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:28.246331    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nwrdf" event={"ID":"3cbbecc2-3a6b-4e03-9060-efc6f1675ae2","Type":"ContainerStarted","Data":"5339f41e9e2c96c5502dfa9c3dfd6fb9b29bf30bb6f97078abec4ed44422d190"}
Jul 07 19:09:28.269450 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:28.269403    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nwrdf" podStartSLOduration=33.576873432 podStartE2EDuration="40.269388761s" totalImagesPullingTime="6.692515329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:08:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:28.268587063 +0000 UTC m=+88.920212996" watchObservedRunningTime="2026-07-07 19:09:28.269388761 +0000 UTC m=+88.921014693"
Jul 07 19:09:31.687179 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.687143    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hc7l5"]
Jul 07 19:09:31.690628 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.690609    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.694211 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.694189    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 19:09:31.694409 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.694386    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 19:09:31.694537 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.694470    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 19:09:31.695372 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.695354    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 19:09:31.695743 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.695726    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 19:09:31.695932 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.695886    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7h7cz\""
Jul 07 19:09:31.696510 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.696484    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 19:09:31.807080 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-wtmp\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807080 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807081    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807128    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-textfile\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-accelerators-collector-config\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-root\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807200    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-metrics-client-ca\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-sys\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807240    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-tls\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.807262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.807256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9sp\" (UniqueName: \"kubernetes.io/projected/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-kube-api-access-nb9sp\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908639 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-textfile\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908639 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-accelerators-collector-config\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908664    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-root\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-metrics-client-ca\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-sys\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-tls\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9sp\" (UniqueName: \"kubernetes.io/projected/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-kube-api-access-nb9sp\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908745    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-root\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908778    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-sys\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908780    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-wtmp\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.908864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.909258 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.908901    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-wtmp\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.909258 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.909089    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-textfile\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.909408 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.909388    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-metrics-client-ca\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.909448 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.909388    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-accelerators-collector-config\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.911097 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.911074    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.911199 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.911167    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-node-exporter-tls\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:31.922660 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:31.922635    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9sp\" (UniqueName: \"kubernetes.io/projected/1f0fb3cb-16c8-4007-a11d-6ec52eb5370f-kube-api-access-nb9sp\") pod \"node-exporter-hc7l5\" (UID: \"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f\") " pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:32.000648 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:32.000606    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hc7l5"
Jul 07 19:09:32.012313 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:09:32.012288    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0fb3cb_16c8_4007_a11d_6ec52eb5370f.slice/crio-9969460c7abba313c98866c793fad5bb80be292b97f00df83e37bce289f2e418 WatchSource:0}: Error finding container 9969460c7abba313c98866c793fad5bb80be292b97f00df83e37bce289f2e418: Status 404 returned error can't find the container with id 9969460c7abba313c98866c793fad5bb80be292b97f00df83e37bce289f2e418
Jul 07 19:09:32.259951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:32.259858    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hc7l5" event={"ID":"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f","Type":"ContainerStarted","Data":"9969460c7abba313c98866c793fad5bb80be292b97f00df83e37bce289f2e418"}
Jul 07 19:09:33.265171 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:33.265138    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hc7l5" event={"ID":"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f","Type":"ContainerStarted","Data":"a41c1e9fca4f84c62e1955d7b1bda84bd1a0534c16a82256f57e317183382256"}
Jul 07 19:09:34.270340 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:34.270303    2567 generic.go:356] "Generic (PLEG): container finished" podID="1f0fb3cb-16c8-4007-a11d-6ec52eb5370f" containerID="a41c1e9fca4f84c62e1955d7b1bda84bd1a0534c16a82256f57e317183382256" exitCode=0
Jul 07 19:09:34.270800 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:34.270359    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hc7l5" event={"ID":"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f","Type":"ContainerDied","Data":"a41c1e9fca4f84c62e1955d7b1bda84bd1a0534c16a82256f57e317183382256"}
Jul 07 19:09:35.174324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:35.174295    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:35.276036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:35.276001    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hc7l5" event={"ID":"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f","Type":"ContainerStarted","Data":"54ab6225194d66e60f44b1af8a6b93b6d08ae267091226cf3fbeb8edea972853"}
Jul 07 19:09:35.276461 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:35.276059    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hc7l5" event={"ID":"1f0fb3cb-16c8-4007-a11d-6ec52eb5370f","Type":"ContainerStarted","Data":"0b394def66338607019c8bf98758069bb82bd3e0de06c6ef4cfab56320636ae8"}
Jul 07 19:09:35.298431 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:35.298389    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hc7l5" podStartSLOduration=3.094376349 podStartE2EDuration="4.298375517s" totalImagesPullingTime="1.140045972s" totalInitContainerRuntime="63.953196ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:35.297464765 +0000 UTC m=+95.949090720" watchObservedRunningTime="2026-07-07 19:09:35.298375517 +0000 UTC m=+95.950001450"
Jul 07 19:09:37.198161 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:37.198134    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b87dd9f8b-zxwpl"
Jul 07 19:09:40.188456 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.188412    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-648d497c84-bppp5" podUID="8f200738-f262-4fc8-9dd5-cf498dc3b77e" containerName="registry" containerID="cri-o://f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e" gracePeriod=30
Jul 07 19:09:40.427300 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.427270    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:40.477796 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477774    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f200738-f262-4fc8-9dd5-cf498dc3b77e-ca-trust-extracted\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.477951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477801    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-trusted-ca\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.477951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477832    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-image-registry-private-configuration\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.477951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477852    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-installation-pull-secrets\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.477951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477867    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-certificates\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.477951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477898    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-bound-sa-token\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.478199 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f55l6\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-kube-api-access-f55l6\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.478199 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.477990    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") pod \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\" (UID: \"8f200738-f262-4fc8-9dd5-cf498dc3b77e\") "
Jul 07 19:09:40.478329 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.478307    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:40.478637 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.478611    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:09:40.480857 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.480828    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:09:40.480996 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.480867    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:09:40.480996 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.480911    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:09:40.480996 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.480945    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-kube-api-access-f55l6" (OuterVolumeSpecName: "kube-api-access-f55l6") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "kube-api-access-f55l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:09:40.481116 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.481051    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:09:40.486929 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.486893    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f200738-f262-4fc8-9dd5-cf498dc3b77e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8f200738-f262-4fc8-9dd5-cf498dc3b77e" (UID: "8f200738-f262-4fc8-9dd5-cf498dc3b77e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:09:40.579204 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579176    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55l6\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-kube-api-access-f55l6\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579204 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579198    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-tls\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579204 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579207    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8f200738-f262-4fc8-9dd5-cf498dc3b77e-ca-trust-extracted\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579386 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579216    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-trusted-ca\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579386 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579225    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-image-registry-private-configuration\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579386 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579234    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8f200738-f262-4fc8-9dd5-cf498dc3b77e-installation-pull-secrets\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579386 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579243    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8f200738-f262-4fc8-9dd5-cf498dc3b77e-registry-certificates\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:40.579386 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:40.579253    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f200738-f262-4fc8-9dd5-cf498dc3b77e-bound-sa-token\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:09:41.160552 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.160524    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pksgb"
Jul 07 19:09:41.297273 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.297233    2567 generic.go:356] "Generic (PLEG): container finished" podID="8f200738-f262-4fc8-9dd5-cf498dc3b77e" containerID="f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e" exitCode=0
Jul 07 19:09:41.297273 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.297276    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-648d497c84-bppp5"
Jul 07 19:09:41.297789 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.297312    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d497c84-bppp5" event={"ID":"8f200738-f262-4fc8-9dd5-cf498dc3b77e","Type":"ContainerDied","Data":"f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e"}
Jul 07 19:09:41.297789 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.297360    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-648d497c84-bppp5" event={"ID":"8f200738-f262-4fc8-9dd5-cf498dc3b77e","Type":"ContainerDied","Data":"e0373a686372f4af50a81783f663c6d3521d3b39d24321d70d5d493be81f2c88"}
Jul 07 19:09:41.297789 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.297380    2567 scope.go:117] "RemoveContainer" containerID="f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e"
Jul 07 19:09:41.306222 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.306205    2567 scope.go:117] "RemoveContainer" containerID="f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e"
Jul 07 19:09:41.306474 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:09:41.306456    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e\": container with ID starting with f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e not found: ID does not exist" containerID="f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e"
Jul 07 19:09:41.306541 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.306483    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e"} err="failed to get container status \"f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e\": rpc error: code = NotFound desc = could not find container \"f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e\": container with ID starting with f80a33c3c520035f4d75bff23a9fd71b8a62104da75647736956ec5a74d88a4e not found: ID does not exist"
Jul 07 19:09:41.330419 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.330397    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-648d497c84-bppp5"]
Jul 07 19:09:41.332712 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.332691    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-648d497c84-bppp5"]
Jul 07 19:09:41.849536 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:09:41.849507    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f200738-f262-4fc8-9dd5-cf498dc3b77e" path="/var/lib/kubelet/pods/8f200738-f262-4fc8-9dd5-cf498dc3b77e/volumes"
Jul 07 19:12:59.681479 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:12:59.681449    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:12:59.685717 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:12:59.685695    2567 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 19:12:59.686297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:12:59.686277    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:14:22.448716 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.448678    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"]
Jul 07 19:14:22.449219 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.448989    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f200738-f262-4fc8-9dd5-cf498dc3b77e" containerName="registry"
Jul 07 19:14:22.449219 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.449008    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f200738-f262-4fc8-9dd5-cf498dc3b77e" containerName="registry"
Jul 07 19:14:22.449219 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.449081    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f200738-f262-4fc8-9dd5-cf498dc3b77e" containerName="registry"
Jul 07 19:14:22.451778 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.451764    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.454251 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.454217    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 07 19:14:22.454380 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.454251    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 07 19:14:22.454380 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.454277    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 07 19:14:22.454380 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.454294    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dfzj8\""
Jul 07 19:14:22.454576 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.454562    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 07 19:14:22.467586 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.467564    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"]
Jul 07 19:14:22.520772 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.520737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.520979 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.520787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.520979 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.520867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-manifests\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.520979 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.520911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-tmp\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.520979 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.520961    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj745\" (UniqueName: \"kubernetes.io/projected/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-kube-api-access-mj745\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.621773 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.621739    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-manifests\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.621773 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.621772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-tmp\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.622047 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.621792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj745\" (UniqueName: \"kubernetes.io/projected/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-kube-api-access-mj745\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.622047 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.621844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.622047 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.621888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.622260 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.622238    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-manifests\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.624254 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.624232    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-tmp\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.624358 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.624288    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.624416 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.624355    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-webhook-cert\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.629569 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.629548    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj745\" (UniqueName: \"kubernetes.io/projected/a961a3a6-ed86-43cf-8678-4d01a77ba4aa-kube-api-access-mj745\") pod \"opendatahub-operator-controller-manager-8689849645-cwbjb\" (UID: \"a961a3a6-ed86-43cf-8678-4d01a77ba4aa\") " pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.762000 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.761971    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:22.924416 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.924386    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"]
Jul 07 19:14:22.926684 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:14:22.926656    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda961a3a6_ed86_43cf_8678_4d01a77ba4aa.slice/crio-9f9b2703f63432d0c5ef7df0f405ccf191d45c3b4da51319bd61389a8cd30f38 WatchSource:0}: Error finding container 9f9b2703f63432d0c5ef7df0f405ccf191d45c3b4da51319bd61389a8cd30f38: Status 404 returned error can't find the container with id 9f9b2703f63432d0c5ef7df0f405ccf191d45c3b4da51319bd61389a8cd30f38
Jul 07 19:14:22.928559 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:22.928541    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:14:23.194771 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:23.194686    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb" event={"ID":"a961a3a6-ed86-43cf-8678-4d01a77ba4aa","Type":"ContainerStarted","Data":"9f9b2703f63432d0c5ef7df0f405ccf191d45c3b4da51319bd61389a8cd30f38"}
Jul 07 19:14:27.215279 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:27.215239    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb" event={"ID":"a961a3a6-ed86-43cf-8678-4d01a77ba4aa","Type":"ContainerStarted","Data":"07f24995455f7ee034a19d057768cb0635972efbe76c0da4769d5968499f0766"}
Jul 07 19:14:28.220341 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:28.220310    2567 generic.go:356] "Generic (PLEG): container finished" podID="a961a3a6-ed86-43cf-8678-4d01a77ba4aa" containerID="07f24995455f7ee034a19d057768cb0635972efbe76c0da4769d5968499f0766" exitCode=0
Jul 07 19:14:28.220958 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:28.220361    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb" event={"ID":"a961a3a6-ed86-43cf-8678-4d01a77ba4aa","Type":"ContainerDied","Data":"07f24995455f7ee034a19d057768cb0635972efbe76c0da4769d5968499f0766"}
Jul 07 19:14:29.224897 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:29.224863    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb" event={"ID":"a961a3a6-ed86-43cf-8678-4d01a77ba4aa","Type":"ContainerStarted","Data":"e8d77c5c22b2db333cc09e381dfff0542652c49176e463d286507f3d02f184ce"}
Jul 07 19:14:29.225286 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:29.224958    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:29.255453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:29.255407    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb" podStartSLOduration=2.235825796 podStartE2EDuration="7.255392524s" totalImagesPullingTime="3.92770608s" totalInitContainerRuntime="1.091860648s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:29.253836332 +0000 UTC m=+389.905462278" watchObservedRunningTime="2026-07-07 19:14:29.255392524 +0000 UTC m=+389.907018456"
Jul 07 19:14:40.231030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:40.230992    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8689849645-cwbjb"
Jul 07 19:14:43.182129 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.182057    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"]
Jul 07 19:14:43.195961 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.195933    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.200431 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.200393    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 07 19:14:43.200575 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.200465    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 07 19:14:43.200575 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.200401    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 07 19:14:43.202940 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.202904    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-88g76\""
Jul 07 19:14:43.204172 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.204153    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 07 19:14:43.209053 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.209030    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"]
Jul 07 19:14:43.376711 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.376682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd66246-00db-4bfa-ad22-f598dff937d4-tls-certs\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.376877 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.376721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cd66246-00db-4bfa-ad22-f598dff937d4-tmp\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.376877 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.376806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsss\" (UniqueName: \"kubernetes.io/projected/9cd66246-00db-4bfa-ad22-f598dff937d4-kube-api-access-zmsss\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.477761 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.477727    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd66246-00db-4bfa-ad22-f598dff937d4-tls-certs\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.477905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.477769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cd66246-00db-4bfa-ad22-f598dff937d4-tmp\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.477905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.477813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsss\" (UniqueName: \"kubernetes.io/projected/9cd66246-00db-4bfa-ad22-f598dff937d4-kube-api-access-zmsss\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.480045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.480023    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cd66246-00db-4bfa-ad22-f598dff937d4-tmp\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.480307 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.480289    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd66246-00db-4bfa-ad22-f598dff937d4-tls-certs\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.501574 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.501552    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsss\" (UniqueName: \"kubernetes.io/projected/9cd66246-00db-4bfa-ad22-f598dff937d4-kube-api-access-zmsss\") pod \"kube-auth-proxy-777c59bd5f-pws4w\" (UID: \"9cd66246-00db-4bfa-ad22-f598dff937d4\") " pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.507408 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.507388    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"
Jul 07 19:14:43.585684 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.585650    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sz472"]
Jul 07 19:14:43.589233 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.589203    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.592416 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.592393    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 07 19:14:43.592534 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.592398    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-x7wqj\""
Jul 07 19:14:43.619117 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.619076    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sz472"]
Jul 07 19:14:43.665558 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.665527    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w"]
Jul 07 19:14:43.668964 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:14:43.668928    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd66246_00db_4bfa_ad22_f598dff937d4.slice/crio-6a25428e5716645ed628aac00591bd80735d19962dd46a925cc689060d2b8eb1 WatchSource:0}: Error finding container 6a25428e5716645ed628aac00591bd80735d19962dd46a925cc689060d2b8eb1: Status 404 returned error can't find the container with id 6a25428e5716645ed628aac00591bd80735d19962dd46a925cc689060d2b8eb1
Jul 07 19:14:43.780605 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.780526    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88debc4-d94c-4551-a194-d792ee4d5117-cert\") pod \"odh-model-controller-7998c67dc4-sz472\" (UID: \"e88debc4-d94c-4551-a194-d792ee4d5117\") " pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.780605 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.780565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxjv\" (UniqueName: \"kubernetes.io/projected/e88debc4-d94c-4551-a194-d792ee4d5117-kube-api-access-cqxjv\") pod \"odh-model-controller-7998c67dc4-sz472\" (UID: \"e88debc4-d94c-4551-a194-d792ee4d5117\") " pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.881524 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.881491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxjv\" (UniqueName: \"kubernetes.io/projected/e88debc4-d94c-4551-a194-d792ee4d5117-kube-api-access-cqxjv\") pod \"odh-model-controller-7998c67dc4-sz472\" (UID: \"e88debc4-d94c-4551-a194-d792ee4d5117\") " pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.881720 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.881569    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88debc4-d94c-4551-a194-d792ee4d5117-cert\") pod \"odh-model-controller-7998c67dc4-sz472\" (UID: \"e88debc4-d94c-4551-a194-d792ee4d5117\") " pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.883887 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.883861    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88debc4-d94c-4551-a194-d792ee4d5117-cert\") pod \"odh-model-controller-7998c67dc4-sz472\" (UID: \"e88debc4-d94c-4551-a194-d792ee4d5117\") " pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.893188 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.893166    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxjv\" (UniqueName: \"kubernetes.io/projected/e88debc4-d94c-4551-a194-d792ee4d5117-kube-api-access-cqxjv\") pod \"odh-model-controller-7998c67dc4-sz472\" (UID: \"e88debc4-d94c-4551-a194-d792ee4d5117\") " pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:43.902927 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:43.902898    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:44.034180 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.034154    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sz472"]
Jul 07 19:14:44.036112 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:14:44.036086    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88debc4_d94c_4551_a194_d792ee4d5117.slice/crio-9f7bebc6f74e19ead3605973166915b11c81c3dac5370990293256c4baae9720 WatchSource:0}: Error finding container 9f7bebc6f74e19ead3605973166915b11c81c3dac5370990293256c4baae9720: Status 404 returned error can't find the container with id 9f7bebc6f74e19ead3605973166915b11c81c3dac5370990293256c4baae9720
Jul 07 19:14:44.275991 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.275954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w" event={"ID":"9cd66246-00db-4bfa-ad22-f598dff937d4","Type":"ContainerStarted","Data":"6a25428e5716645ed628aac00591bd80735d19962dd46a925cc689060d2b8eb1"}
Jul 07 19:14:44.276931 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.276887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sz472" event={"ID":"e88debc4-d94c-4551-a194-d792ee4d5117","Type":"ContainerStarted","Data":"9f7bebc6f74e19ead3605973166915b11c81c3dac5370990293256c4baae9720"}
Jul 07 19:14:44.646447 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.646399    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wxxct"]
Jul 07 19:14:44.649991 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.649970    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:44.652698 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.652669    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 07 19:14:44.652820 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.652706    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-k96bm\""
Jul 07 19:14:44.662566 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.662544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wxxct"]
Jul 07 19:14:44.787115 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.787078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrf6l\" (UniqueName: \"kubernetes.io/projected/7f298848-7828-4177-8d19-09b8a4954e27-kube-api-access-wrf6l\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:44.787292 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.787161    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f298848-7828-4177-8d19-09b8a4954e27-tls-certs\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:44.844687 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.844648    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"]
Jul 07 19:14:44.848035 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.848009    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.850835 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.850810    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 07 19:14:44.851132 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.851058    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 07 19:14:44.851132 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.851065    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6kwhl\""
Jul 07 19:14:44.851411 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.851387    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 07 19:14:44.851498 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.851397    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 07 19:14:44.865977 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.865951    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"]
Jul 07 19:14:44.888146 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.888097    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrf6l\" (UniqueName: \"kubernetes.io/projected/7f298848-7828-4177-8d19-09b8a4954e27-kube-api-access-wrf6l\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:44.888272 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.888169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f298848-7828-4177-8d19-09b8a4954e27-tls-certs\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:44.888336 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:14:44.888286    2567 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 07 19:14:44.888397 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:14:44.888379    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f298848-7828-4177-8d19-09b8a4954e27-tls-certs podName:7f298848-7828-4177-8d19-09b8a4954e27 nodeName:}" failed. No retries permitted until 2026-07-07 19:14:45.388358081 +0000 UTC m=+406.039984006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7f298848-7828-4177-8d19-09b8a4954e27-tls-certs") pod "model-serving-api-54d89bd5fc-wxxct" (UID: "7f298848-7828-4177-8d19-09b8a4954e27") : secret "model-serving-api-tls" not found
Jul 07 19:14:44.899722 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.899659    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrf6l\" (UniqueName: \"kubernetes.io/projected/7f298848-7828-4177-8d19-09b8a4954e27-kube-api-access-wrf6l\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:44.989236 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftcg\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-kube-api-access-5ftcg\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.989236 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989147    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.989236 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-cacerts\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.989665 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989255    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-local-certs\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.989665 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989297    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-token\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.989665 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:44.989665 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:44.989364    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090647 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-local-certs\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090827 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090659    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-token\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090827 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090827 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090827 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftcg\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-kube-api-access-5ftcg\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090827 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.090827 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.090779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-cacerts\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.091502 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.091476    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.093175 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.093147    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.093310 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.093285    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-cacerts\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.093376 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.093341    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-local-certs\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.094084 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.093871    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.100785 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.100761    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftcg\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-kube-api-access-5ftcg\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.102222 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.102197    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-token\") pod \"istiod-openshift-gateway-7574459656-k9x4v\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.159969 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.159892    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:45.316235 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.316209    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"]
Jul 07 19:14:45.318322 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:14:45.318290    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9314ae03_cdd2_4951_b2f7_0d7e74517f3e.slice/crio-94193c64301cbca46795dc502e2ff6ae9504ee22ed2200477acb0b998fd2ad5d WatchSource:0}: Error finding container 94193c64301cbca46795dc502e2ff6ae9504ee22ed2200477acb0b998fd2ad5d: Status 404 returned error can't find the container with id 94193c64301cbca46795dc502e2ff6ae9504ee22ed2200477acb0b998fd2ad5d
Jul 07 19:14:45.393270 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.393235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f298848-7828-4177-8d19-09b8a4954e27-tls-certs\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:45.395945 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.395905    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f298848-7828-4177-8d19-09b8a4954e27-tls-certs\") pod \"model-serving-api-54d89bd5fc-wxxct\" (UID: \"7f298848-7828-4177-8d19-09b8a4954e27\") " pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:45.563430 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.563352    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:45.724737 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:45.724710    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-wxxct"]
Jul 07 19:14:45.727284 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:14:45.727254    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f298848_7828_4177_8d19_09b8a4954e27.slice/crio-d5cc5ae7bc40170bdbbeb1bc4a1f9192c690cec93b30ea7a681be0b9bd87e74a WatchSource:0}: Error finding container d5cc5ae7bc40170bdbbeb1bc4a1f9192c690cec93b30ea7a681be0b9bd87e74a: Status 404 returned error can't find the container with id d5cc5ae7bc40170bdbbeb1bc4a1f9192c690cec93b30ea7a681be0b9bd87e74a
Jul 07 19:14:46.290866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:46.290768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct" event={"ID":"7f298848-7828-4177-8d19-09b8a4954e27","Type":"ContainerStarted","Data":"d5cc5ae7bc40170bdbbeb1bc4a1f9192c690cec93b30ea7a681be0b9bd87e74a"}
Jul 07 19:14:46.292857 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:46.292799    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" event={"ID":"9314ae03-cdd2-4951-b2f7-0d7e74517f3e","Type":"ContainerStarted","Data":"94193c64301cbca46795dc502e2ff6ae9504ee22ed2200477acb0b998fd2ad5d"}
Jul 07 19:14:49.667794 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:49.667753    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 19:14:49.668136 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:49.667871    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 07 19:14:50.311282 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.311234    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sz472" event={"ID":"e88debc4-d94c-4551-a194-d792ee4d5117","Type":"ContainerStarted","Data":"0b06c53615fef37383c56af76fa8bad16742935c652aabed481f290c27a9fb97"}
Jul 07 19:14:50.312048 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.312020    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:14:50.313744 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.313707    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct" event={"ID":"7f298848-7828-4177-8d19-09b8a4954e27","Type":"ContainerStarted","Data":"0b37bdf62affb0ed792b92a6c54dfc858074d68d93b209d4579ece8e052174a2"}
Jul 07 19:14:50.314246 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.314226    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:14:50.316112 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.316084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" event={"ID":"9314ae03-cdd2-4951-b2f7-0d7e74517f3e","Type":"ContainerStarted","Data":"5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97"}
Jul 07 19:14:50.316319 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.316302    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:50.317901 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.317873    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w" event={"ID":"9cd66246-00db-4bfa-ad22-f598dff937d4","Type":"ContainerStarted","Data":"ad6ee564affe65635cd687323923f00e85c7e3ba1a83960fb793754dad069ab1"}
Jul 07 19:14:50.318323 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.318287    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-k9x4v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 19:14:50.318417 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.318340    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 19:14:50.337847 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.337801    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sz472" podStartSLOduration=1.698055461 podStartE2EDuration="7.337786564s" totalImagesPullingTime="5.639731103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:50.336736689 +0000 UTC m=+410.988362624" watchObservedRunningTime="2026-07-07 19:14:50.337786564 +0000 UTC m=+410.989412494"
Jul 07 19:14:50.369778 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.369736    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct" podStartSLOduration=2.428538424 podStartE2EDuration="6.369724026s" totalImagesPullingTime="3.941185602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:50.36810454 +0000 UTC m=+411.019730473" watchObservedRunningTime="2026-07-07 19:14:50.369724026 +0000 UTC m=+411.021349959"
Jul 07 19:14:50.402791 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.402743    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" podStartSLOduration=2.055946431 podStartE2EDuration="6.402727498s" totalImagesPullingTime="4.346781067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:50.40212172 +0000 UTC m=+411.053747653" watchObservedRunningTime="2026-07-07 19:14:50.402727498 +0000 UTC m=+411.054353431"
Jul 07 19:14:50.424872 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:50.424828    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-777c59bd5f-pws4w" podStartSLOduration=1.428406984 podStartE2EDuration="7.42481478s" totalImagesPullingTime="5.996407796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:50.423702278 +0000 UTC m=+411.075328233" watchObservedRunningTime="2026-07-07 19:14:50.42481478 +0000 UTC m=+411.076440713"
Jul 07 19:14:51.322466 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:51.322430    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-k9x4v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 07 19:14:51.322865 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:51.322486    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 19:14:53.602676 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.602641    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"]
Jul 07 19:14:53.605835 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.605819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.609932 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.609888    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 19:14:53.610052 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.609955    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 19:14:53.611374 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.611356    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rm5xg\""
Jul 07 19:14:53.611482 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.611465    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 19:14:53.612124 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.612106    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 19:14:53.614256 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.614239    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 19:14:53.636225 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.636199    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"]
Jul 07 19:14:53.653287 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.653260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c1fea3b-36df-4217-865b-8b8c1bdfed23-cert\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.653395 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.653299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c1fea3b-36df-4217-865b-8b8c1bdfed23-metrics-cert\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.653395 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.653334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2c1fea3b-36df-4217-865b-8b8c1bdfed23-manager-config\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.653469 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.653401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpdc6\" (UniqueName: \"kubernetes.io/projected/2c1fea3b-36df-4217-865b-8b8c1bdfed23-kube-api-access-cpdc6\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.754419 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.754379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c1fea3b-36df-4217-865b-8b8c1bdfed23-metrics-cert\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.754562 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.754426    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2c1fea3b-36df-4217-865b-8b8c1bdfed23-manager-config\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.754562 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.754462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpdc6\" (UniqueName: \"kubernetes.io/projected/2c1fea3b-36df-4217-865b-8b8c1bdfed23-kube-api-access-cpdc6\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.754562 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.754530    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c1fea3b-36df-4217-865b-8b8c1bdfed23-cert\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.755207 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.755182    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2c1fea3b-36df-4217-865b-8b8c1bdfed23-manager-config\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.757190 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.757162    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c1fea3b-36df-4217-865b-8b8c1bdfed23-metrics-cert\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.757267 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.757169    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c1fea3b-36df-4217-865b-8b8c1bdfed23-cert\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.765731 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.765709    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpdc6\" (UniqueName: \"kubernetes.io/projected/2c1fea3b-36df-4217-865b-8b8c1bdfed23-kube-api-access-cpdc6\") pod \"lws-controller-manager-5d95ccd586-j7c9m\" (UID: \"2c1fea3b-36df-4217-865b-8b8c1bdfed23\") " pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:53.915677 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:53.915594    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:54.090083 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:14:54.090055    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1fea3b_36df_4217_865b_8b8c1bdfed23.slice/crio-2376c556385146e9d9e9136d61402b6963930f7d16b52675905f5240633a46ea WatchSource:0}: Error finding container 2376c556385146e9d9e9136d61402b6963930f7d16b52675905f5240633a46ea: Status 404 returned error can't find the container with id 2376c556385146e9d9e9136d61402b6963930f7d16b52675905f5240633a46ea
Jul 07 19:14:54.091489 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:54.091465    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"]
Jul 07 19:14:54.323194 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:54.323159    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:54.333983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:54.333954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m" event={"ID":"2c1fea3b-36df-4217-865b-8b8c1bdfed23","Type":"ContainerStarted","Data":"2376c556385146e9d9e9136d61402b6963930f7d16b52675905f5240633a46ea"}
Jul 07 19:14:57.262013 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.261976    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"]
Jul 07 19:14:57.262419 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.262164    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery" containerID="cri-o://5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97" gracePeriod=30
Jul 07 19:14:57.322794 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.322762    2567 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-k9x4v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.19:8080/ready\": dial tcp 10.132.0.19:8080: connect: connection refused" start-of-body=
Jul 07 19:14:57.322905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.322813    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.19:8080/ready\": dial tcp 10.132.0.19:8080: connect: connection refused"
Jul 07 19:14:57.349345 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.349318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m" event={"ID":"2c1fea3b-36df-4217-865b-8b8c1bdfed23","Type":"ContainerStarted","Data":"88458c7c14b32ac79a785aea0efabebab0466b0a6aae61e7e3be614ca828020c"}
Jul 07 19:14:57.349480 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.349461    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:14:57.372380 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.372339    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m" podStartSLOduration=2.029097892 podStartE2EDuration="4.372324183s" totalImagesPullingTime="2.343226291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:14:57.371163687 +0000 UTC m=+418.022789624" watchObservedRunningTime="2026-07-07 19:14:57.372324183 +0000 UTC m=+418.023950115"
Jul 07 19:14:57.515696 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.515643    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:57.594888 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.594857    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-kubeconfig\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.594911    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-token\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.594962    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-ca-configmap\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.594996    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ftcg\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-kube-api-access-5ftcg\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.595022    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-cacerts\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595241 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.595085    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-local-certs\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595241 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.595113    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-dns-cert\") pod \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\" (UID: \"9314ae03-cdd2-4951-b2f7-0d7e74517f3e\") "
Jul 07 19:14:57.595472 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.595439    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 19:14:57.597380 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.597348    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:14:57.597472 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.597406    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:14:57.597556 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.597539    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-kube-api-access-5ftcg" (OuterVolumeSpecName: "kube-api-access-5ftcg") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "kube-api-access-5ftcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:14:57.597609 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.597584    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-local-certs" (OuterVolumeSpecName: "local-certs") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:14:57.597642 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.597615    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-cacerts" (OuterVolumeSpecName: "cacerts") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:14:57.597895 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.597869    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9314ae03-cdd2-4951-b2f7-0d7e74517f3e" (UID: "9314ae03-cdd2-4951-b2f7-0d7e74517f3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:14:57.696674 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696638    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-token\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:57.696674 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696667    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-ca-configmap\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:57.696674 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696678    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ftcg\" (UniqueName: \"kubernetes.io/projected/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-kube-api-access-5ftcg\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:57.696872 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696688    2567 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-cacerts\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:57.696872 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696697    2567 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-local-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:57.696872 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696705    2567 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-csr-dns-cert\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:57.696872 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:57.696712    2567 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9314ae03-cdd2-4951-b2f7-0d7e74517f3e-istio-kubeconfig\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:14:58.354013 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.353980    2567 generic.go:356] "Generic (PLEG): container finished" podID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerID="5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97" exitCode=0
Jul 07 19:14:58.354396 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.354035    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"
Jul 07 19:14:58.354396 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.354071    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" event={"ID":"9314ae03-cdd2-4951-b2f7-0d7e74517f3e","Type":"ContainerDied","Data":"5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97"}
Jul 07 19:14:58.354396 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.354125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v" event={"ID":"9314ae03-cdd2-4951-b2f7-0d7e74517f3e","Type":"ContainerDied","Data":"94193c64301cbca46795dc502e2ff6ae9504ee22ed2200477acb0b998fd2ad5d"}
Jul 07 19:14:58.354396 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.354147    2567 scope.go:117] "RemoveContainer" containerID="5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97"
Jul 07 19:14:58.362146 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.362128    2567 scope.go:117] "RemoveContainer" containerID="5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97"
Jul 07 19:14:58.362416 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:14:58.362400    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97\": container with ID starting with 5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97 not found: ID does not exist" containerID="5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97"
Jul 07 19:14:58.362479 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.362426    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97"} err="failed to get container status \"5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97\": rpc error: code = NotFound desc = could not find container \"5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97\": container with ID starting with 5d719a4ae78365bd7cb5665f31783c5e9cee6bc49aa42939151a50dfb853fe97 not found: ID does not exist"
Jul 07 19:14:58.400781 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.400756    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"]
Jul 07 19:14:58.415329 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:58.415301    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k9x4v"]
Jul 07 19:14:59.851145 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:14:59.851117    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" path="/var/lib/kubelet/pods/9314ae03-cdd2-4951-b2f7-0d7e74517f3e/volumes"
Jul 07 19:15:02.327859 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:15:02.327829    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sz472"
Jul 07 19:15:02.329863 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:15:02.329845    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-wxxct"
Jul 07 19:15:08.356964 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:15:08.356914    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d95ccd586-j7c9m"
Jul 07 19:16:31.454368 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.454327    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"]
Jul 07 19:16:31.454814 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.454720    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery"
Jul 07 19:16:31.454814 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.454736    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery"
Jul 07 19:16:31.454814 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.454808    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9314ae03-cdd2-4951-b2f7-0d7e74517f3e" containerName="discovery"
Jul 07 19:16:31.457906 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.457887    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.460776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.460751    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 07 19:16:31.460897 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.460760    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6d87x\""
Jul 07 19:16:31.460897 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.460762    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 19:16:31.461652 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.461632    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 19:16:31.461719 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.461632    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 07 19:16:31.467292 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.467270    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"]
Jul 07 19:16:31.485012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.484983    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7586f04-2b32-4bc1-816f-fcd58e784f98-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.485117 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.485082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5xv\" (UniqueName: \"kubernetes.io/projected/b7586f04-2b32-4bc1-816f-fcd58e784f98-kube-api-access-6t5xv\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.485117 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.485108    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7586f04-2b32-4bc1-816f-fcd58e784f98-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.586189 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.586156    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7586f04-2b32-4bc1-816f-fcd58e784f98-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.586391 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.586232    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5xv\" (UniqueName: \"kubernetes.io/projected/b7586f04-2b32-4bc1-816f-fcd58e784f98-kube-api-access-6t5xv\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.586391 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.586257    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7586f04-2b32-4bc1-816f-fcd58e784f98-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.586391 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:16:31.586322    2567 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 07 19:16:31.586391 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:16:31.586381    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7586f04-2b32-4bc1-816f-fcd58e784f98-plugin-serving-cert podName:b7586f04-2b32-4bc1-816f-fcd58e784f98 nodeName:}" failed. No retries permitted until 2026-07-07 19:16:32.086361963 +0000 UTC m=+512.737987887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b7586f04-2b32-4bc1-816f-fcd58e784f98-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-kd7sm" (UID: "b7586f04-2b32-4bc1-816f-fcd58e784f98") : secret "plugin-serving-cert" not found
Jul 07 19:16:31.586891 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.586870    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7586f04-2b32-4bc1-816f-fcd58e784f98-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:31.595812 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:31.595781    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5xv\" (UniqueName: \"kubernetes.io/projected/b7586f04-2b32-4bc1-816f-fcd58e784f98-kube-api-access-6t5xv\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:32.089573 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:32.089538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7586f04-2b32-4bc1-816f-fcd58e784f98-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:32.091808 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:32.091772    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7586f04-2b32-4bc1-816f-fcd58e784f98-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kd7sm\" (UID: \"b7586f04-2b32-4bc1-816f-fcd58e784f98\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:32.369010 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:32.368931    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"
Jul 07 19:16:32.499290 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:32.499259    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm"]
Jul 07 19:16:32.503087 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:16:32.503063    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7586f04_2b32_4bc1_816f_fcd58e784f98.slice/crio-4f5a8ea14f472a2eddd9ee48383c3bccc1fc6b3733ea394a189b612c3a1584b0 WatchSource:0}: Error finding container 4f5a8ea14f472a2eddd9ee48383c3bccc1fc6b3733ea394a189b612c3a1584b0: Status 404 returned error can't find the container with id 4f5a8ea14f472a2eddd9ee48383c3bccc1fc6b3733ea394a189b612c3a1584b0
Jul 07 19:16:32.684025 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:32.683946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm" event={"ID":"b7586f04-2b32-4bc1-816f-fcd58e784f98","Type":"ContainerStarted","Data":"4f5a8ea14f472a2eddd9ee48383c3bccc1fc6b3733ea394a189b612c3a1584b0"}
Jul 07 19:16:57.798459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:57.798423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm" event={"ID":"b7586f04-2b32-4bc1-816f-fcd58e784f98","Type":"ContainerStarted","Data":"ae3877d6f18ae16ed51135a37710ec5153e4f01c2a84fe53ca1a30e912b1a342"}
Jul 07 19:16:57.815931 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:16:57.815871    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kd7sm" podStartSLOduration=2.28039428 podStartE2EDuration="26.815853725s" totalImagesPullingTime="24.535459445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:16:57.81538962 +0000 UTC m=+538.467015555" watchObservedRunningTime="2026-07-07 19:16:57.815853725 +0000 UTC m=+538.467479659"
Jul 07 19:17:14.178857 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.178819    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-grfdg"]
Jul 07 19:17:14.241139 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.241099    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-grfdg"]
Jul 07 19:17:14.241329 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.241187    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:14.244936 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.244893    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-2ng4m\""
Jul 07 19:17:14.339173 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.339141    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlhmv\" (UniqueName: \"kubernetes.io/projected/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f-kube-api-access-xlhmv\") pod \"authorino-58cd955675-grfdg\" (UID: \"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f\") " pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:14.440028 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.439945    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlhmv\" (UniqueName: \"kubernetes.io/projected/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f-kube-api-access-xlhmv\") pod \"authorino-58cd955675-grfdg\" (UID: \"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f\") " pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:14.456277 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.456252    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlhmv\" (UniqueName: \"kubernetes.io/projected/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f-kube-api-access-xlhmv\") pod \"authorino-58cd955675-grfdg\" (UID: \"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f\") " pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:14.551216 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.551189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:14.560888 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.560861    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gfrw9"]
Jul 07 19:17:14.564454 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.564432    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:14.574435 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.574410    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gfrw9"]
Jul 07 19:17:14.679977 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.679949    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-grfdg"]
Jul 07 19:17:14.682277 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:17:14.682252    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd3290a2_35a6_4ad6_b930_11e1d85ecb7f.slice/crio-3addac5aa9d0a42e4ffbd4387507fd00778576eef2f0a06b5986af1bf75f7b06 WatchSource:0}: Error finding container 3addac5aa9d0a42e4ffbd4387507fd00778576eef2f0a06b5986af1bf75f7b06: Status 404 returned error can't find the container with id 3addac5aa9d0a42e4ffbd4387507fd00778576eef2f0a06b5986af1bf75f7b06
Jul 07 19:17:14.743790 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.743758    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pds9w\" (UniqueName: \"kubernetes.io/projected/ac29fdc0-9d7f-4231-b1f2-246f9a565b42-kube-api-access-pds9w\") pod \"authorino-75d556bd77-gfrw9\" (UID: \"ac29fdc0-9d7f-4231-b1f2-246f9a565b42\") " pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:14.844598 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.844561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pds9w\" (UniqueName: \"kubernetes.io/projected/ac29fdc0-9d7f-4231-b1f2-246f9a565b42-kube-api-access-pds9w\") pod \"authorino-75d556bd77-gfrw9\" (UID: \"ac29fdc0-9d7f-4231-b1f2-246f9a565b42\") " pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:14.853511 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.853486    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pds9w\" (UniqueName: \"kubernetes.io/projected/ac29fdc0-9d7f-4231-b1f2-246f9a565b42-kube-api-access-pds9w\") pod \"authorino-75d556bd77-gfrw9\" (UID: \"ac29fdc0-9d7f-4231-b1f2-246f9a565b42\") " pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:14.863892 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.863868    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grfdg" event={"ID":"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f","Type":"ContainerStarted","Data":"3addac5aa9d0a42e4ffbd4387507fd00778576eef2f0a06b5986af1bf75f7b06"}
Jul 07 19:17:14.881222 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:14.881202    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:15.005036 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:15.005012    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gfrw9"]
Jul 07 19:17:15.007363 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:17:15.007340    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac29fdc0_9d7f_4231_b1f2_246f9a565b42.slice/crio-2362db555f26b7e41e94f35e9e2a8c9c11c4289a10578bd8f1ecf9206139c489 WatchSource:0}: Error finding container 2362db555f26b7e41e94f35e9e2a8c9c11c4289a10578bd8f1ecf9206139c489: Status 404 returned error can't find the container with id 2362db555f26b7e41e94f35e9e2a8c9c11c4289a10578bd8f1ecf9206139c489
Jul 07 19:17:15.877526 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:15.877449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gfrw9" event={"ID":"ac29fdc0-9d7f-4231-b1f2-246f9a565b42","Type":"ContainerStarted","Data":"2362db555f26b7e41e94f35e9e2a8c9c11c4289a10578bd8f1ecf9206139c489"}
Jul 07 19:17:18.896068 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:18.895970    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grfdg" event={"ID":"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f","Type":"ContainerStarted","Data":"edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3"}
Jul 07 19:17:18.897328 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:18.897295    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gfrw9" event={"ID":"ac29fdc0-9d7f-4231-b1f2-246f9a565b42","Type":"ContainerStarted","Data":"b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a"}
Jul 07 19:17:18.949634 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:18.949583    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gfrw9" podStartSLOduration=1.43904891 podStartE2EDuration="4.949566843s" totalImagesPullingTime="3.510517933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:18.949261215 +0000 UTC m=+559.600887167" watchObservedRunningTime="2026-07-07 19:17:18.949566843 +0000 UTC m=+559.601192775"
Jul 07 19:17:18.950078 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:18.950048    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-grfdg" podStartSLOduration=1.124863578 podStartE2EDuration="4.950039095s" totalImagesPullingTime="3.825175517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:18.927061092 +0000 UTC m=+559.578687025" watchObservedRunningTime="2026-07-07 19:17:18.950039095 +0000 UTC m=+559.601665081"
Jul 07 19:17:18.996467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:18.996437    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-grfdg"]
Jul 07 19:17:20.904452 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:20.904412    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-grfdg" podUID="cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" containerName="authorino" containerID="cri-o://edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3" gracePeriod=30
Jul 07 19:17:21.153523 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.153502    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:21.297959 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.297913    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlhmv\" (UniqueName: \"kubernetes.io/projected/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f-kube-api-access-xlhmv\") pod \"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f\" (UID: \"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f\") "
Jul 07 19:17:21.300042 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.300013    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f-kube-api-access-xlhmv" (OuterVolumeSpecName: "kube-api-access-xlhmv") pod "cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" (UID: "cd3290a2-35a6-4ad6-b930-11e1d85ecb7f"). InnerVolumeSpecName "kube-api-access-xlhmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:17:21.399314 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.399288    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlhmv\" (UniqueName: \"kubernetes.io/projected/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f-kube-api-access-xlhmv\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:17:21.909352 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.909322    2567 generic.go:356] "Generic (PLEG): container finished" podID="cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" containerID="edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3" exitCode=0
Jul 07 19:17:21.909777 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.909361    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-grfdg"
Jul 07 19:17:21.909777 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.909388    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grfdg" event={"ID":"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f","Type":"ContainerDied","Data":"edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3"}
Jul 07 19:17:21.909777 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.909426    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-grfdg" event={"ID":"cd3290a2-35a6-4ad6-b930-11e1d85ecb7f","Type":"ContainerDied","Data":"3addac5aa9d0a42e4ffbd4387507fd00778576eef2f0a06b5986af1bf75f7b06"}
Jul 07 19:17:21.909777 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.909451    2567 scope.go:117] "RemoveContainer" containerID="edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3"
Jul 07 19:17:21.917651 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.917634    2567 scope.go:117] "RemoveContainer" containerID="edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3"
Jul 07 19:17:21.917946 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:17:21.917905    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3\": container with ID starting with edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3 not found: ID does not exist" containerID="edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3"
Jul 07 19:17:21.918020 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.917956    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3"} err="failed to get container status \"edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3\": rpc error: code = NotFound desc = could not find container \"edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3\": container with ID starting with edc3648165ad3fb2cec6cc5b5fa4d5c932405075f917c6c8c82360f6cf25e7d3 not found: ID does not exist"
Jul 07 19:17:21.924530 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.924506    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-grfdg"]
Jul 07 19:17:21.928892 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:21.928872    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-grfdg"]
Jul 07 19:17:23.850439 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:23.850412    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" path="/var/lib/kubelet/pods/cd3290a2-35a6-4ad6-b930-11e1d85ecb7f/volumes"
Jul 07 19:17:46.920469 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:46.920433    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9btk9"]
Jul 07 19:17:46.920896 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:46.920831    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" containerName="authorino"
Jul 07 19:17:46.920896 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:46.920845    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" containerName="authorino"
Jul 07 19:17:46.921000 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:46.920913    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3290a2-35a6-4ad6-b930-11e1d85ecb7f" containerName="authorino"
Jul 07 19:17:46.979898 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:46.979871    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9btk9"]
Jul 07 19:17:46.980046 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:46.979982    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:47.081166 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.081135    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwhz\" (UniqueName: \"kubernetes.io/projected/c419f326-ed9f-40e1-adb0-816bb5f393e0-kube-api-access-vlwhz\") pod \"authorino-5d4b67c446-9btk9\" (UID: \"c419f326-ed9f-40e1-adb0-816bb5f393e0\") " pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:47.169401 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.169367    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9btk9"]
Jul 07 19:17:47.169597 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:17:47.169580    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vlwhz], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-9btk9" podUID="c419f326-ed9f-40e1-adb0-816bb5f393e0"
Jul 07 19:17:47.182399 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.182347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwhz\" (UniqueName: \"kubernetes.io/projected/c419f326-ed9f-40e1-adb0-816bb5f393e0-kube-api-access-vlwhz\") pod \"authorino-5d4b67c446-9btk9\" (UID: \"c419f326-ed9f-40e1-adb0-816bb5f393e0\") " pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:47.198032 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.198001    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwhz\" (UniqueName: \"kubernetes.io/projected/c419f326-ed9f-40e1-adb0-816bb5f393e0-kube-api-access-vlwhz\") pod \"authorino-5d4b67c446-9btk9\" (UID: \"c419f326-ed9f-40e1-adb0-816bb5f393e0\") " pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:47.204634 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.204610    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-644d98bb4d-8c6j8"]
Jul 07 19:17:47.207876 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.207862    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:47.224388 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.224363    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-644d98bb4d-8c6j8"]
Jul 07 19:17:47.283623 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.283590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7md\" (UniqueName: \"kubernetes.io/projected/e764c599-933b-4dff-b50e-503496a7d604-kube-api-access-sj7md\") pod \"authorino-644d98bb4d-8c6j8\" (UID: \"e764c599-933b-4dff-b50e-503496a7d604\") " pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:47.384907 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.384877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7md\" (UniqueName: \"kubernetes.io/projected/e764c599-933b-4dff-b50e-503496a7d604-kube-api-access-sj7md\") pod \"authorino-644d98bb4d-8c6j8\" (UID: \"e764c599-933b-4dff-b50e-503496a7d604\") " pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:47.404479 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.404454    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7md\" (UniqueName: \"kubernetes.io/projected/e764c599-933b-4dff-b50e-503496a7d604-kube-api-access-sj7md\") pod \"authorino-644d98bb4d-8c6j8\" (UID: \"e764c599-933b-4dff-b50e-503496a7d604\") " pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:47.451146 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.451081    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-644d98bb4d-8c6j8"]
Jul 07 19:17:47.451298 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.451285    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:47.508178 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.508145    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c47f69bd-hqlvw"]
Jul 07 19:17:47.512228 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.512201    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.516977 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.516952    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 19:17:47.531140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.531114    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c47f69bd-hqlvw"]
Jul 07 19:17:47.586379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.586352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbf2\" (UniqueName: \"kubernetes.io/projected/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-kube-api-access-hxbf2\") pod \"authorino-59c47f69bd-hqlvw\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") " pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.586520 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.586393    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-tls-cert\") pod \"authorino-59c47f69bd-hqlvw\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") " pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.610201 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.610172    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-644d98bb4d-8c6j8"]
Jul 07 19:17:47.612441 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:17:47.612404    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode764c599_933b_4dff_b50e_503496a7d604.slice/crio-54ca67b751bbffb7c24be93231892dcc24adde8c914fc3ee04b382297c150af5 WatchSource:0}: Error finding container 54ca67b751bbffb7c24be93231892dcc24adde8c914fc3ee04b382297c150af5: Status 404 returned error can't find the container with id 54ca67b751bbffb7c24be93231892dcc24adde8c914fc3ee04b382297c150af5
Jul 07 19:17:47.687437 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.687409    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-tls-cert\") pod \"authorino-59c47f69bd-hqlvw\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") " pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.687568 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.687478    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbf2\" (UniqueName: \"kubernetes.io/projected/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-kube-api-access-hxbf2\") pod \"authorino-59c47f69bd-hqlvw\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") " pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.689706 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.689688    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-tls-cert\") pod \"authorino-59c47f69bd-hqlvw\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") " pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.697159 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.697134    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbf2\" (UniqueName: \"kubernetes.io/projected/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-kube-api-access-hxbf2\") pod \"authorino-59c47f69bd-hqlvw\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") " pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.824935 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.824894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:17:47.956257 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:47.956164    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c47f69bd-hqlvw"]
Jul 07 19:17:47.958589 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:17:47.958562    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc2f4e43_f3b6_4460_9d9c_214fe28eb88f.slice/crio-ec31f09ec5b49414c67d5166bb2d1fa406e33d77a9b55dd8984232f92d59bb0d WatchSource:0}: Error finding container ec31f09ec5b49414c67d5166bb2d1fa406e33d77a9b55dd8984232f92d59bb0d: Status 404 returned error can't find the container with id ec31f09ec5b49414c67d5166bb2d1fa406e33d77a9b55dd8984232f92d59bb0d
Jul 07 19:17:48.008337 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.008306    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644d98bb4d-8c6j8" event={"ID":"e764c599-933b-4dff-b50e-503496a7d604","Type":"ContainerStarted","Data":"54ca67b751bbffb7c24be93231892dcc24adde8c914fc3ee04b382297c150af5"}
Jul 07 19:17:48.009683 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.009663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:48.009803 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.009660    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c47f69bd-hqlvw" event={"ID":"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f","Type":"ContainerStarted","Data":"ec31f09ec5b49414c67d5166bb2d1fa406e33d77a9b55dd8984232f92d59bb0d"}
Jul 07 19:17:48.016337 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.016320    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:48.091113 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.091093    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlwhz\" (UniqueName: \"kubernetes.io/projected/c419f326-ed9f-40e1-adb0-816bb5f393e0-kube-api-access-vlwhz\") pod \"c419f326-ed9f-40e1-adb0-816bb5f393e0\" (UID: \"c419f326-ed9f-40e1-adb0-816bb5f393e0\") "
Jul 07 19:17:48.092941 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.092909    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c419f326-ed9f-40e1-adb0-816bb5f393e0-kube-api-access-vlwhz" (OuterVolumeSpecName: "kube-api-access-vlwhz") pod "c419f326-ed9f-40e1-adb0-816bb5f393e0" (UID: "c419f326-ed9f-40e1-adb0-816bb5f393e0"). InnerVolumeSpecName "kube-api-access-vlwhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:17:48.192117 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:48.192073    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlwhz\" (UniqueName: \"kubernetes.io/projected/c419f326-ed9f-40e1-adb0-816bb5f393e0-kube-api-access-vlwhz\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:17:49.014608 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.014571    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644d98bb4d-8c6j8" event={"ID":"e764c599-933b-4dff-b50e-503496a7d604","Type":"ContainerStarted","Data":"59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d"}
Jul 07 19:17:49.014608 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.014603    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-644d98bb4d-8c6j8" podUID="e764c599-933b-4dff-b50e-503496a7d604" containerName="authorino" containerID="cri-o://59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d" gracePeriod=30
Jul 07 19:17:49.016071 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.016050    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-9btk9"
Jul 07 19:17:49.016180 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.016049    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c47f69bd-hqlvw" event={"ID":"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f","Type":"ContainerStarted","Data":"f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a"}
Jul 07 19:17:49.040796 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.040755    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-644d98bb4d-8c6j8" podStartSLOduration=1.6167811890000001 podStartE2EDuration="2.040741409s" totalImagesPullingTime="423.96022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:49.039355068 +0000 UTC m=+589.690981012" watchObservedRunningTime="2026-07-07 19:17:49.040741409 +0000 UTC m=+589.692367376"
Jul 07 19:17:49.122698 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.122643    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c47f69bd-hqlvw" podStartSLOduration=1.760548597 podStartE2EDuration="2.122624175s" totalImagesPullingTime="362.075578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:17:49.089666793 +0000 UTC m=+589.741292728" watchObservedRunningTime="2026-07-07 19:17:49.122624175 +0000 UTC m=+589.774250114"
Jul 07 19:17:49.138261 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.138227    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gfrw9"]
Jul 07 19:17:49.138470 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.138443    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gfrw9" podUID="ac29fdc0-9d7f-4231-b1f2-246f9a565b42" containerName="authorino" containerID="cri-o://b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a" gracePeriod=30
Jul 07 19:17:49.146486 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.146463    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9btk9"]
Jul 07 19:17:49.167045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.166980    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-9btk9"]
Jul 07 19:17:49.265783 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.265729    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:49.302421 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.302396    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj7md\" (UniqueName: \"kubernetes.io/projected/e764c599-933b-4dff-b50e-503496a7d604-kube-api-access-sj7md\") pod \"e764c599-933b-4dff-b50e-503496a7d604\" (UID: \"e764c599-933b-4dff-b50e-503496a7d604\") "
Jul 07 19:17:49.304723 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.304692    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e764c599-933b-4dff-b50e-503496a7d604-kube-api-access-sj7md" (OuterVolumeSpecName: "kube-api-access-sj7md") pod "e764c599-933b-4dff-b50e-503496a7d604" (UID: "e764c599-933b-4dff-b50e-503496a7d604"). InnerVolumeSpecName "kube-api-access-sj7md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:17:49.384905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.384885    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:49.403199 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.403179    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7md\" (UniqueName: \"kubernetes.io/projected/e764c599-933b-4dff-b50e-503496a7d604-kube-api-access-sj7md\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:17:49.503411 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.503383    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pds9w\" (UniqueName: \"kubernetes.io/projected/ac29fdc0-9d7f-4231-b1f2-246f9a565b42-kube-api-access-pds9w\") pod \"ac29fdc0-9d7f-4231-b1f2-246f9a565b42\" (UID: \"ac29fdc0-9d7f-4231-b1f2-246f9a565b42\") "
Jul 07 19:17:49.505318 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.505287    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac29fdc0-9d7f-4231-b1f2-246f9a565b42-kube-api-access-pds9w" (OuterVolumeSpecName: "kube-api-access-pds9w") pod "ac29fdc0-9d7f-4231-b1f2-246f9a565b42" (UID: "ac29fdc0-9d7f-4231-b1f2-246f9a565b42"). InnerVolumeSpecName "kube-api-access-pds9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:17:49.604601 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.604539    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pds9w\" (UniqueName: \"kubernetes.io/projected/ac29fdc0-9d7f-4231-b1f2-246f9a565b42-kube-api-access-pds9w\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:17:49.849518 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:49.849485    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c419f326-ed9f-40e1-adb0-816bb5f393e0" path="/var/lib/kubelet/pods/c419f326-ed9f-40e1-adb0-816bb5f393e0/volumes"
Jul 07 19:17:50.021417 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.021378    2567 generic.go:356] "Generic (PLEG): container finished" podID="ac29fdc0-9d7f-4231-b1f2-246f9a565b42" containerID="b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a" exitCode=0
Jul 07 19:17:50.021866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.021426    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gfrw9"
Jul 07 19:17:50.021866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.021529    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gfrw9" event={"ID":"ac29fdc0-9d7f-4231-b1f2-246f9a565b42","Type":"ContainerDied","Data":"b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a"}
Jul 07 19:17:50.021866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.021575    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gfrw9" event={"ID":"ac29fdc0-9d7f-4231-b1f2-246f9a565b42","Type":"ContainerDied","Data":"2362db555f26b7e41e94f35e9e2a8c9c11c4289a10578bd8f1ecf9206139c489"}
Jul 07 19:17:50.021866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.021600    2567 scope.go:117] "RemoveContainer" containerID="b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a"
Jul 07 19:17:50.022970 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.022772    2567 generic.go:356] "Generic (PLEG): container finished" podID="e764c599-933b-4dff-b50e-503496a7d604" containerID="59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d" exitCode=0
Jul 07 19:17:50.022970 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.022809    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-644d98bb4d-8c6j8"
Jul 07 19:17:50.022970 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.022833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644d98bb4d-8c6j8" event={"ID":"e764c599-933b-4dff-b50e-503496a7d604","Type":"ContainerDied","Data":"59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d"}
Jul 07 19:17:50.022970 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.022859    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-644d98bb4d-8c6j8" event={"ID":"e764c599-933b-4dff-b50e-503496a7d604","Type":"ContainerDied","Data":"54ca67b751bbffb7c24be93231892dcc24adde8c914fc3ee04b382297c150af5"}
Jul 07 19:17:50.029655 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.029639    2567 scope.go:117] "RemoveContainer" containerID="b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a"
Jul 07 19:17:50.029907 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:17:50.029885    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a\": container with ID starting with b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a not found: ID does not exist" containerID="b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a"
Jul 07 19:17:50.030004 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.029911    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a"} err="failed to get container status \"b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a\": rpc error: code = NotFound desc = could not find container \"b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a\": container with ID starting with b0af389c301afa9a6250ec7ca8b0550b7fea5eedf96fb40eb3f1af98cd03ab5a not found: ID does not exist"
Jul 07 19:17:50.030004 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.029943    2567 scope.go:117] "RemoveContainer" containerID="59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d"
Jul 07 19:17:50.036849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.036820    2567 scope.go:117] "RemoveContainer" containerID="59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d"
Jul 07 19:17:50.037105 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:17:50.037083    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d\": container with ID starting with 59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d not found: ID does not exist" containerID="59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d"
Jul 07 19:17:50.037191 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.037111    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d"} err="failed to get container status \"59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d\": rpc error: code = NotFound desc = could not find container \"59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d\": container with ID starting with 59bed66efa352fdae7bcfcf4b8aec94eaef37646642af66bd6d46025fedb7a1d not found: ID does not exist"
Jul 07 19:17:50.067368 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.067346    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-644d98bb4d-8c6j8"]
Jul 07 19:17:50.085399 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.085377    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-644d98bb4d-8c6j8"]
Jul 07 19:17:50.105589 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.105568    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gfrw9"]
Jul 07 19:17:50.110681 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:50.110631    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gfrw9"]
Jul 07 19:17:51.849894 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:51.849865    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac29fdc0-9d7f-4231-b1f2-246f9a565b42" path="/var/lib/kubelet/pods/ac29fdc0-9d7f-4231-b1f2-246f9a565b42/volumes"
Jul 07 19:17:51.850286 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:51.850201    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e764c599-933b-4dff-b50e-503496a7d604" path="/var/lib/kubelet/pods/e764c599-933b-4dff-b50e-503496a7d604/volumes"
Jul 07 19:17:59.705598 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:59.705567    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:17:59.707665 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:17:59.707643    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:18:30.772140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772063    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sz4zv"]
Jul 07 19:18:30.772583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772326    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e764c599-933b-4dff-b50e-503496a7d604" containerName="authorino"
Jul 07 19:18:30.772583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772337    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e764c599-933b-4dff-b50e-503496a7d604" containerName="authorino"
Jul 07 19:18:30.772583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772354    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac29fdc0-9d7f-4231-b1f2-246f9a565b42" containerName="authorino"
Jul 07 19:18:30.772583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772360    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac29fdc0-9d7f-4231-b1f2-246f9a565b42" containerName="authorino"
Jul 07 19:18:30.772583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772406    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac29fdc0-9d7f-4231-b1f2-246f9a565b42" containerName="authorino"
Jul 07 19:18:30.772583 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.772414    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e764c599-933b-4dff-b50e-503496a7d604" containerName="authorino"
Jul 07 19:18:30.775178 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.775161    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:30.777707 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.777687    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 07 19:18:30.804799 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.804773    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sz4zv"]
Jul 07 19:18:30.815720 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.815694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfw47\" (UniqueName: \"kubernetes.io/projected/e740858b-e6ec-47b8-892f-c70d7c647cea-kube-api-access-zfw47\") pod \"payload-processing-58f86c7798-sz4zv\" (UID: \"e740858b-e6ec-47b8-892f-c70d7c647cea\") " pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:30.815828 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.815747    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e740858b-e6ec-47b8-892f-c70d7c647cea-plugins-config-volume\") pod \"payload-processing-58f86c7798-sz4zv\" (UID: \"e740858b-e6ec-47b8-892f-c70d7c647cea\") " pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:30.916280 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.916250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfw47\" (UniqueName: \"kubernetes.io/projected/e740858b-e6ec-47b8-892f-c70d7c647cea-kube-api-access-zfw47\") pod \"payload-processing-58f86c7798-sz4zv\" (UID: \"e740858b-e6ec-47b8-892f-c70d7c647cea\") " pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:30.916444 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.916291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e740858b-e6ec-47b8-892f-c70d7c647cea-plugins-config-volume\") pod \"payload-processing-58f86c7798-sz4zv\" (UID: \"e740858b-e6ec-47b8-892f-c70d7c647cea\") " pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:30.916895 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.916876    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e740858b-e6ec-47b8-892f-c70d7c647cea-plugins-config-volume\") pod \"payload-processing-58f86c7798-sz4zv\" (UID: \"e740858b-e6ec-47b8-892f-c70d7c647cea\") " pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:30.935907 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:30.935882    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfw47\" (UniqueName: \"kubernetes.io/projected/e740858b-e6ec-47b8-892f-c70d7c647cea-kube-api-access-zfw47\") pod \"payload-processing-58f86c7798-sz4zv\" (UID: \"e740858b-e6ec-47b8-892f-c70d7c647cea\") " pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:31.085552 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:31.085473    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:31.218025 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:31.217996    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-sz4zv"]
Jul 07 19:18:31.221424 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:18:31.221399    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode740858b_e6ec_47b8_892f_c70d7c647cea.slice/crio-23a3c710594601059b4ee6050eac3afcedf043c99d884823ec985d342968681a WatchSource:0}: Error finding container 23a3c710594601059b4ee6050eac3afcedf043c99d884823ec985d342968681a: Status 404 returned error can't find the container with id 23a3c710594601059b4ee6050eac3afcedf043c99d884823ec985d342968681a
Jul 07 19:18:32.195267 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:32.195227    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv" event={"ID":"e740858b-e6ec-47b8-892f-c70d7c647cea","Type":"ContainerStarted","Data":"23a3c710594601059b4ee6050eac3afcedf043c99d884823ec985d342968681a"}
Jul 07 19:18:33.200058 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:33.199970    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv" event={"ID":"e740858b-e6ec-47b8-892f-c70d7c647cea","Type":"ContainerStarted","Data":"6aaad237996314114dbe9fb779c62558ad72e5e7e62c07972b1256905c2de1da"}
Jul 07 19:18:33.200441 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:33.200117    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:18:33.223799 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:33.223751    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv" podStartSLOduration=1.719192286 podStartE2EDuration="3.223737167s" totalImagesPullingTime="1.504544881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:18:33.223081838 +0000 UTC m=+633.874707771" watchObservedRunningTime="2026-07-07 19:18:33.223737167 +0000 UTC m=+633.875363100"
Jul 07 19:18:44.206056 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:18:44.206027    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-sz4zv"
Jul 07 19:19:09.778206 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.778174    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"]
Jul 07 19:19:09.781568 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.781548    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:09.785499 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.785480    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 07 19:19:09.785767 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.785749    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 07 19:19:09.785842 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.785772    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 07 19:19:09.785842 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.785753    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n7c64\""
Jul 07 19:19:09.795388 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.795366    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"]
Jul 07 19:19:09.911167 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.911137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:09.911167 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.911170    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f740a4-4b87-4f2a-960f-3b810fb5ff66-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:09.911394 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.911189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:09.911394 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.911209    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:09.911394 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.911273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d947s\" (UniqueName: \"kubernetes.io/projected/67f740a4-4b87-4f2a-960f-3b810fb5ff66-kube-api-access-d947s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:09.911394 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:09.911374    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012550 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012520    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012695 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d947s\" (UniqueName: \"kubernetes.io/projected/67f740a4-4b87-4f2a-960f-3b810fb5ff66-kube-api-access-d947s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012695 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012810 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012755    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012810 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012784    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f740a4-4b87-4f2a-960f-3b810fb5ff66-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012912 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.012912 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.012846    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.013107 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.013086    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.013205 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.013174    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.014968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.014936    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f740a4-4b87-4f2a-960f-3b810fb5ff66-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.015180 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.015160    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f740a4-4b87-4f2a-960f-3b810fb5ff66-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.024476 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.024454    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d947s\" (UniqueName: \"kubernetes.io/projected/67f740a4-4b87-4f2a-960f-3b810fb5ff66-kube-api-access-d947s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7nfvn\" (UID: \"67f740a4-4b87-4f2a-960f-3b810fb5ff66\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.091350 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.091293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:10.236155 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.236126    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"]
Jul 07 19:19:10.237966 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:19:10.237942    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f740a4_4b87_4f2a_960f_3b810fb5ff66.slice/crio-2b1133d7da097b6b5b88bb55e610cfe7ad9001af02838353479f3be16e2a6945 WatchSource:0}: Error finding container 2b1133d7da097b6b5b88bb55e610cfe7ad9001af02838353479f3be16e2a6945: Status 404 returned error can't find the container with id 2b1133d7da097b6b5b88bb55e610cfe7ad9001af02838353479f3be16e2a6945
Jul 07 19:19:10.347279 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:10.347205    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn" event={"ID":"67f740a4-4b87-4f2a-960f-3b810fb5ff66","Type":"ContainerStarted","Data":"2b1133d7da097b6b5b88bb55e610cfe7ad9001af02838353479f3be16e2a6945"}
Jul 07 19:19:11.353045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:11.353016    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn" event={"ID":"67f740a4-4b87-4f2a-960f-3b810fb5ff66","Type":"ContainerStarted","Data":"25499d3833f16ef736fea769686556bcb991acf9866e95e56e9ba412f1c57733"}
Jul 07 19:19:11.377347 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:11.377299    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn" podStartSLOduration=1.347062507 podStartE2EDuration="2.377284339s" totalImagesPullingTime="1.030221832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:19:11.375444904 +0000 UTC m=+672.027070837" watchObservedRunningTime="2026-07-07 19:19:11.377284339 +0000 UTC m=+672.028910272"
Jul 07 19:19:20.058586 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.058547    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"]
Jul 07 19:19:20.061899 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.061881    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.065602 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.065582    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 07 19:19:20.075967 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.075946    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"]
Jul 07 19:19:20.091486 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.091462    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:20.091579 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.091506    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:20.103793 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.103764    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:20.191733 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.191702    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.191733 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.191736    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.191947 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.191756    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.191947 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.191788    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b417a3-7e78-41ef-bc7a-281d02b4b467-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.191947 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.191835    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.191947 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.191911    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmgj\" (UniqueName: \"kubernetes.io/projected/d1b417a3-7e78-41ef-bc7a-281d02b4b467-kube-api-access-jsmgj\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293260 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293373 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293265    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293373 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b417a3-7e78-41ef-bc7a-281d02b4b467-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293373 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293332    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293504 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293382    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmgj\" (UniqueName: \"kubernetes.io/projected/d1b417a3-7e78-41ef-bc7a-281d02b4b467-kube-api-access-jsmgj\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293504 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293731 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293707    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293844 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293798    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.293890 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.293859    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.295587 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.295570    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b417a3-7e78-41ef-bc7a-281d02b4b467-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.295700 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.295682    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b417a3-7e78-41ef-bc7a-281d02b4b467-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.302590 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.302566    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmgj\" (UniqueName: \"kubernetes.io/projected/d1b417a3-7e78-41ef-bc7a-281d02b4b467-kube-api-access-jsmgj\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q\" (UID: \"d1b417a3-7e78-41ef-bc7a-281d02b4b467\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.370477 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.370416    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:20.401324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.401289    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7nfvn"
Jul 07 19:19:20.506348 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:20.506260    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"]
Jul 07 19:19:20.508772 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:19:20.508743    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1b417a3_7e78_41ef_bc7a_281d02b4b467.slice/crio-6e118b53ed227904febd82e15897d8fe6e934f92f280a248e1688c5b185d3929 WatchSource:0}: Error finding container 6e118b53ed227904febd82e15897d8fe6e934f92f280a248e1688c5b185d3929: Status 404 returned error can't find the container with id 6e118b53ed227904febd82e15897d8fe6e934f92f280a248e1688c5b185d3929
Jul 07 19:19:21.394839 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:21.394798    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q" event={"ID":"d1b417a3-7e78-41ef-bc7a-281d02b4b467","Type":"ContainerStarted","Data":"7b717c770b00541670949671723a7b8e1e1d5a0d77aefd9eabeeef2ed39bcc99"}
Jul 07 19:19:21.394839 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:21.394844    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q" event={"ID":"d1b417a3-7e78-41ef-bc7a-281d02b4b467","Type":"ContainerStarted","Data":"6e118b53ed227904febd82e15897d8fe6e934f92f280a248e1688c5b185d3929"}
Jul 07 19:19:21.417480 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:21.417437    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q" podStartSLOduration=1.229152735 podStartE2EDuration="1.417423481s" totalImagesPullingTime="188.270746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:19:21.41565649 +0000 UTC m=+682.067282423" watchObservedRunningTime="2026-07-07 19:19:21.417423481 +0000 UTC m=+682.069049413"
Jul 07 19:19:30.370885 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:30.370855    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:30.370885 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:30.370896    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:30.383152 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:30.383130    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:19:30.440441 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:19:30.440411    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q"
Jul 07 19:20:28.093550 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.093516    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7444d9f889-z8wqf"]
Jul 07 19:20:28.096689 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.096673    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.107404 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.107382    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7444d9f889-z8wqf"]
Jul 07 19:20:28.233748 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.233720    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eec2c6d1-42d7-4e07-b7bb-82875e16a5ae-tls-cert\") pod \"authorino-7444d9f889-z8wqf\" (UID: \"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae\") " pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.233880 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.233761    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2c5\" (UniqueName: \"kubernetes.io/projected/eec2c6d1-42d7-4e07-b7bb-82875e16a5ae-kube-api-access-lc2c5\") pod \"authorino-7444d9f889-z8wqf\" (UID: \"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae\") " pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.334579 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.334544    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eec2c6d1-42d7-4e07-b7bb-82875e16a5ae-tls-cert\") pod \"authorino-7444d9f889-z8wqf\" (UID: \"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae\") " pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.334710 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.334591    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2c5\" (UniqueName: \"kubernetes.io/projected/eec2c6d1-42d7-4e07-b7bb-82875e16a5ae-kube-api-access-lc2c5\") pod \"authorino-7444d9f889-z8wqf\" (UID: \"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae\") " pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.336954 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.336935    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eec2c6d1-42d7-4e07-b7bb-82875e16a5ae-tls-cert\") pod \"authorino-7444d9f889-z8wqf\" (UID: \"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae\") " pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.353794 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.353738    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2c5\" (UniqueName: \"kubernetes.io/projected/eec2c6d1-42d7-4e07-b7bb-82875e16a5ae-kube-api-access-lc2c5\") pod \"authorino-7444d9f889-z8wqf\" (UID: \"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae\") " pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.405391 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.405369    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7444d9f889-z8wqf"
Jul 07 19:20:28.538317 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.538279    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7444d9f889-z8wqf"]
Jul 07 19:20:28.539012 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:20:28.538987    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec2c6d1_42d7_4e07_b7bb_82875e16a5ae.slice/crio-0bdcc5995f8df4d5ac25993562f2db6f8b67543b7f58a63ab32284a12f16cb3b WatchSource:0}: Error finding container 0bdcc5995f8df4d5ac25993562f2db6f8b67543b7f58a63ab32284a12f16cb3b: Status 404 returned error can't find the container with id 0bdcc5995f8df4d5ac25993562f2db6f8b67543b7f58a63ab32284a12f16cb3b
Jul 07 19:20:28.540634 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.540613    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:20:28.664749 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:28.664675    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7444d9f889-z8wqf" event={"ID":"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae","Type":"ContainerStarted","Data":"0bdcc5995f8df4d5ac25993562f2db6f8b67543b7f58a63ab32284a12f16cb3b"}
Jul 07 19:20:29.670227 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:29.670192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7444d9f889-z8wqf" event={"ID":"eec2c6d1-42d7-4e07-b7bb-82875e16a5ae","Type":"ContainerStarted","Data":"e9feac172a48236babf24ea0ab30476cd43bd11f7b6b5b3baac0f84147496542"}
Jul 07 19:20:29.711482 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:29.711419    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7444d9f889-z8wqf" podStartSLOduration=1.323714303 podStartE2EDuration="1.711393466s" totalImagesPullingTime="387.679163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:20:29.709239777 +0000 UTC m=+750.360865711" watchObservedRunningTime="2026-07-07 19:20:29.711393466 +0000 UTC m=+750.363019403"
Jul 07 19:20:29.771555 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:29.771519    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c47f69bd-hqlvw"]
Jul 07 19:20:29.771710 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:29.771691    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-59c47f69bd-hqlvw" podUID="dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" containerName="authorino" containerID="cri-o://f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a" gracePeriod=30
Jul 07 19:20:30.031981 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.031953    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:20:30.152262 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.152228    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxbf2\" (UniqueName: \"kubernetes.io/projected/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-kube-api-access-hxbf2\") pod \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") "
Jul 07 19:20:30.152446 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.152287    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-tls-cert\") pod \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\" (UID: \"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f\") "
Jul 07 19:20:30.154319 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.154286    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-kube-api-access-hxbf2" (OuterVolumeSpecName: "kube-api-access-hxbf2") pod "dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" (UID: "dc2f4e43-f3b6-4460-9d9c-214fe28eb88f"). InnerVolumeSpecName "kube-api-access-hxbf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:20:30.162022 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.161996    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" (UID: "dc2f4e43-f3b6-4460-9d9c-214fe28eb88f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:20:30.252946 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.252897    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-tls-cert\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:20:30.252946 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.252942    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbf2\" (UniqueName: \"kubernetes.io/projected/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f-kube-api-access-hxbf2\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:20:30.675796 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.675692    2567 generic.go:356] "Generic (PLEG): container finished" podID="dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" containerID="f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a" exitCode=0
Jul 07 19:20:30.675796 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.675745    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c47f69bd-hqlvw"
Jul 07 19:20:30.675796 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.675774    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c47f69bd-hqlvw" event={"ID":"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f","Type":"ContainerDied","Data":"f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a"}
Jul 07 19:20:30.676308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.675822    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c47f69bd-hqlvw" event={"ID":"dc2f4e43-f3b6-4460-9d9c-214fe28eb88f","Type":"ContainerDied","Data":"ec31f09ec5b49414c67d5166bb2d1fa406e33d77a9b55dd8984232f92d59bb0d"}
Jul 07 19:20:30.676308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.675842    2567 scope.go:117] "RemoveContainer" containerID="f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a"
Jul 07 19:20:30.685121 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.685106    2567 scope.go:117] "RemoveContainer" containerID="f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a"
Jul 07 19:20:30.685389 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:20:30.685371    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a\": container with ID starting with f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a not found: ID does not exist" containerID="f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a"
Jul 07 19:20:30.685456 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.685400    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a"} err="failed to get container status \"f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a\": rpc error: code = NotFound desc = could not find container \"f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a\": container with ID starting with f8039eafbec1563b53407beba66b31c21e5d355a05f42dd395391c960405363a not found: ID does not exist"
Jul 07 19:20:30.709714 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.709687    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c47f69bd-hqlvw"]
Jul 07 19:20:30.714393 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:30.714370    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c47f69bd-hqlvw"]
Jul 07 19:20:31.850503 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:20:31.850462    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" path="/var/lib/kubelet/pods/dc2f4e43-f3b6-4460-9d9c-214fe28eb88f/volumes"
Jul 07 19:22:59.732401 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:22:59.732372    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:22:59.732879 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:22:59.732470    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:27:59.754559 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:27:59.754526    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:27:59.756303 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:27:59.756282    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:32:59.776799 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:32:59.776772    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:32:59.779264 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:32:59.779244    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:37:23.262446 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.262411    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"]
Jul 07 19:37:23.262984 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.262722    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" containerName="authorino"
Jul 07 19:37:23.262984 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.262733    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" containerName="authorino"
Jul 07 19:37:23.262984 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.262795    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc2f4e43-f3b6-4460-9d9c-214fe28eb88f" containerName="authorino"
Jul 07 19:37:23.265680 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.265664    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.269304 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.269280    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-adb1679d\"/\"default-dockercfg-4t4bq\""
Jul 07 19:37:23.269445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.269280    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-adb1679d\"/\"openshift-service-ca.crt\""
Jul 07 19:37:23.269445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.269282    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-adb1679d\"/\"e2e-lifecycle-model-adb1679d-kserve-self-signed-certs\""
Jul 07 19:37:23.269445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.269343    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-adb1679d\"/\"kube-root-ca.crt\""
Jul 07 19:37:23.274492 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.274468    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"]
Jul 07 19:37:23.416074 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.416035    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-dshm\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.416074 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.416074    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c2bfd8-36de-4a4f-aead-f112a53f7051-tls-certs\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.416296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.416098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6d8j\" (UniqueName: \"kubernetes.io/projected/98c2bfd8-36de-4a4f-aead-f112a53f7051-kube-api-access-z6d8j\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.416296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.416183    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-tmp-dir\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.416296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.416232    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-model-cache\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.416296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.416277    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-home\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517090    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-model-cache\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-home\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-dshm\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517438 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c2bfd8-36de-4a4f-aead-f112a53f7051-tls-certs\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517438 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517212    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6d8j\" (UniqueName: \"kubernetes.io/projected/98c2bfd8-36de-4a4f-aead-f112a53f7051-kube-api-access-z6d8j\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517438 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-tmp-dir\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517567    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-model-cache\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517683 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517667    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-home\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.517751 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.517722    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-tmp-dir\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.519513 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.519489    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-dshm\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.519766 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.519748    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c2bfd8-36de-4a4f-aead-f112a53f7051-tls-certs\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.525911 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.525887    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6d8j\" (UniqueName: \"kubernetes.io/projected/98c2bfd8-36de-4a4f-aead-f112a53f7051-kube-api-access-z6d8j\") pod \"e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") " pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.575622 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.575589    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:23.704533 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.704509    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"]
Jul 07 19:37:23.707386 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:37:23.707358    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98c2bfd8_36de_4a4f_aead_f112a53f7051.slice/crio-e103fee6a499e0f525d13aeccf74e86f0602c24550f7ed9e2e4d105e994c8fb4 WatchSource:0}: Error finding container e103fee6a499e0f525d13aeccf74e86f0602c24550f7ed9e2e4d105e994c8fb4: Status 404 returned error can't find the container with id e103fee6a499e0f525d13aeccf74e86f0602c24550f7ed9e2e4d105e994c8fb4
Jul 07 19:37:23.730349 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:23.730325    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj" event={"ID":"98c2bfd8-36de-4a4f-aead-f112a53f7051","Type":"ContainerStarted","Data":"e103fee6a499e0f525d13aeccf74e86f0602c24550f7ed9e2e4d105e994c8fb4"}
Jul 07 19:37:24.736087 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:24.736050    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj" event={"ID":"98c2bfd8-36de-4a4f-aead-f112a53f7051","Type":"ContainerStarted","Data":"3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc"}
Jul 07 19:37:24.754346 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:24.754303    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj" podStartSLOduration=1.754288474 podStartE2EDuration="1.754288474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:37:24.753383365 +0000 UTC m=+1765.405009298" watchObservedRunningTime="2026-07-07 19:37:24.754288474 +0000 UTC m=+1765.405914407"
Jul 07 19:37:33.576319 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:33.576285    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:33.576319 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:33.576326    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:33.588862 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:33.588841    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:33.782468 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:33.782440    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:37:59.807458 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:59.807428    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:37:59.811396 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:37:59.811376    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:38:11.564978 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:11.564941    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"]
Jul 07 19:38:26.584811 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.584767    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj" podUID="98c2bfd8-36de-4a4f-aead-f112a53f7051" containerName="main" containerID="cri-o://3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc" gracePeriod=45
Jul 07 19:38:26.830608 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.830582    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:38:26.843106 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843046    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-dshm\") pod \"98c2bfd8-36de-4a4f-aead-f112a53f7051\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") "
Jul 07 19:38:26.843106 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843084    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-model-cache\") pod \"98c2bfd8-36de-4a4f-aead-f112a53f7051\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") "
Jul 07 19:38:26.843272 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843141    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6d8j\" (UniqueName: \"kubernetes.io/projected/98c2bfd8-36de-4a4f-aead-f112a53f7051-kube-api-access-z6d8j\") pod \"98c2bfd8-36de-4a4f-aead-f112a53f7051\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") "
Jul 07 19:38:26.843272 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843180    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c2bfd8-36de-4a4f-aead-f112a53f7051-tls-certs\") pod \"98c2bfd8-36de-4a4f-aead-f112a53f7051\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") "
Jul 07 19:38:26.843272 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843215    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-tmp-dir\") pod \"98c2bfd8-36de-4a4f-aead-f112a53f7051\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") "
Jul 07 19:38:26.843429 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843331    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-home\") pod \"98c2bfd8-36de-4a4f-aead-f112a53f7051\" (UID: \"98c2bfd8-36de-4a4f-aead-f112a53f7051\") "
Jul 07 19:38:26.843429 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843410    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-model-cache" (OuterVolumeSpecName: "model-cache") pod "98c2bfd8-36de-4a4f-aead-f112a53f7051" (UID: "98c2bfd8-36de-4a4f-aead-f112a53f7051"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:38:26.843526 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843443    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98c2bfd8-36de-4a4f-aead-f112a53f7051" (UID: "98c2bfd8-36de-4a4f-aead-f112a53f7051"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:38:26.843604 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843582    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-home" (OuterVolumeSpecName: "home") pod "98c2bfd8-36de-4a4f-aead-f112a53f7051" (UID: "98c2bfd8-36de-4a4f-aead-f112a53f7051"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:38:26.843657 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843639    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:38:26.843657 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843652    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:38:26.843722 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.843660    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:38:26.845354 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.845331    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c2bfd8-36de-4a4f-aead-f112a53f7051-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98c2bfd8-36de-4a4f-aead-f112a53f7051" (UID: "98c2bfd8-36de-4a4f-aead-f112a53f7051"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:38:26.845573 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.845547    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-dshm" (OuterVolumeSpecName: "dshm") pod "98c2bfd8-36de-4a4f-aead-f112a53f7051" (UID: "98c2bfd8-36de-4a4f-aead-f112a53f7051"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:38:26.845573 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.845560    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c2bfd8-36de-4a4f-aead-f112a53f7051-kube-api-access-z6d8j" (OuterVolumeSpecName: "kube-api-access-z6d8j") pod "98c2bfd8-36de-4a4f-aead-f112a53f7051" (UID: "98c2bfd8-36de-4a4f-aead-f112a53f7051"). InnerVolumeSpecName "kube-api-access-z6d8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:38:26.944820 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.944789    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98c2bfd8-36de-4a4f-aead-f112a53f7051-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:38:26.944820 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.944813    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6d8j\" (UniqueName: \"kubernetes.io/projected/98c2bfd8-36de-4a4f-aead-f112a53f7051-kube-api-access-z6d8j\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:38:26.944820 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.944822    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98c2bfd8-36de-4a4f-aead-f112a53f7051-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:38:26.983314 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.983281    2567 generic.go:356] "Generic (PLEG): container finished" podID="98c2bfd8-36de-4a4f-aead-f112a53f7051" containerID="3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc" exitCode=0
Jul 07 19:38:26.983445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.983326    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"
Jul 07 19:38:26.983445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.983361    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj" event={"ID":"98c2bfd8-36de-4a4f-aead-f112a53f7051","Type":"ContainerDied","Data":"3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc"}
Jul 07 19:38:26.983445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.983398    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj" event={"ID":"98c2bfd8-36de-4a4f-aead-f112a53f7051","Type":"ContainerDied","Data":"e103fee6a499e0f525d13aeccf74e86f0602c24550f7ed9e2e4d105e994c8fb4"}
Jul 07 19:38:26.983445 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.983419    2567 scope.go:117] "RemoveContainer" containerID="3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc"
Jul 07 19:38:26.991874 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.991855    2567 scope.go:117] "RemoveContainer" containerID="3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc"
Jul 07 19:38:26.992132 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:38:26.992111    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc\": container with ID starting with 3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc not found: ID does not exist" containerID="3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc"
Jul 07 19:38:26.992196 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:26.992138    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc"} err="failed to get container status \"3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc\": rpc error: code = NotFound desc = could not find container \"3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc\": container with ID starting with 3187d12c3a68c246d1fff12b7af16e1a4cbdf0b44a066d238518a5f57c7705dc not found: ID does not exist"
Jul 07 19:38:27.006409 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:27.006383    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"]
Jul 07 19:38:27.010452 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:27.010430    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-adb1679d/e2e-lifecycle-model-adb1679d-kserve-65cb669d8c-6dbwj"]
Jul 07 19:38:27.850770 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:38:27.850735    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98c2bfd8-36de-4a4f-aead-f112a53f7051" path="/var/lib/kubelet/pods/98c2bfd8-36de-4a4f-aead-f112a53f7051/volumes"
Jul 07 19:40:07.829978 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.829946    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"]
Jul 07 19:40:07.830453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.830244    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98c2bfd8-36de-4a4f-aead-f112a53f7051" containerName="main"
Jul 07 19:40:07.830453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.830254    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="98c2bfd8-36de-4a4f-aead-f112a53f7051" containerName="main"
Jul 07 19:40:07.830453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.830300    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="98c2bfd8-36de-4a4f-aead-f112a53f7051" containerName="main"
Jul 07 19:40:07.833123 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.833105    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:07.839099 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.839064    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-444f1d\"/\"kube-root-ca.crt\""
Jul 07 19:40:07.839239 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.839140    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-444f1d\"/\"default-dockercfg-8wlxk\""
Jul 07 19:40:07.839239 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.839197    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-444f1d\"/\"openshift-service-ca.crt\""
Jul 07 19:40:07.839239 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.839206    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-444f1d\"/\"test-model-444f1d-kserve-self-signed-certs\""
Jul 07 19:40:07.850354 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.850335    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"]
Jul 07 19:40:07.967844 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.967816    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-dshm\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:07.968021 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.967861    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-tmp-dir\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:07.968021 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.967885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736c32f9-4828-4f85-a94b-ff90174429e9-tls-certs\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:07.968021 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.967970    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-home\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:07.968021 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.968004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvg5v\" (UniqueName: \"kubernetes.io/projected/736c32f9-4828-4f85-a94b-ff90174429e9-kube-api-access-lvg5v\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:07.968178 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:07.968029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-model-cache\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069212 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069178    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvg5v\" (UniqueName: \"kubernetes.io/projected/736c32f9-4828-4f85-a94b-ff90174429e9-kube-api-access-lvg5v\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069212 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069217    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-model-cache\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069378 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069240    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-dshm\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069378 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069290    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-tmp-dir\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069378 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736c32f9-4828-4f85-a94b-ff90174429e9-tls-certs\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069378 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-home\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069657 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069638    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-model-cache\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069758 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069736    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-home\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.069758 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.069732    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-tmp-dir\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.071570 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.071544    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-dshm\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.071730 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.071715    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736c32f9-4828-4f85-a94b-ff90174429e9-tls-certs\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.079010 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.078988    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvg5v\" (UniqueName: \"kubernetes.io/projected/736c32f9-4828-4f85-a94b-ff90174429e9-kube-api-access-lvg5v\") pod \"test-model-444f1d-kserve-6b567ffcd7-5f59t\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") " pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.148604 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.148543    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:08.280141 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.280092    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"]
Jul 07 19:40:08.282205 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:40:08.282178    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod736c32f9_4828_4f85_a94b_ff90174429e9.slice/crio-0036e99ced0840dfc97f382321263db3e7e3f219f85e0cb3ee6159029974d330 WatchSource:0}: Error finding container 0036e99ced0840dfc97f382321263db3e7e3f219f85e0cb3ee6159029974d330: Status 404 returned error can't find the container with id 0036e99ced0840dfc97f382321263db3e7e3f219f85e0cb3ee6159029974d330
Jul 07 19:40:08.405300 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.405226    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t" event={"ID":"736c32f9-4828-4f85-a94b-ff90174429e9","Type":"ContainerStarted","Data":"b427cf77c71e0df46e8887991b9e0feb005f0afdee3e5ab7c06ed3f6c81f4fe0"}
Jul 07 19:40:08.405300 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.405272    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t" event={"ID":"736c32f9-4828-4f85-a94b-ff90174429e9","Type":"ContainerStarted","Data":"0036e99ced0840dfc97f382321263db3e7e3f219f85e0cb3ee6159029974d330"}
Jul 07 19:40:08.428880 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:08.428834    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t" podStartSLOduration=1.428819951 podStartE2EDuration="1.428819951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:40:08.42832238 +0000 UTC m=+1929.079948314" watchObservedRunningTime="2026-07-07 19:40:08.428819951 +0000 UTC m=+1929.080445883"
Jul 07 19:40:18.149428 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:18.149390    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:18.151759 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:18.149529    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:18.162017 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:18.161994    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:18.463233 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:18.463162    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:40:40.184653 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.184621    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"]
Jul 07 19:40:40.188173 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.188155    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.191120 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.191098    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-548bc4\"/\"openshift-service-ca.crt\""
Jul 07 19:40:40.191250 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.191118    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-548bc4\"/\"kube-root-ca.crt\""
Jul 07 19:40:40.191964 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.191950    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-548bc4\"/\"default-dockercfg-26pbn\""
Jul 07 19:40:40.192067 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.192051    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-548bc4\"/\"test-model-548bc4-kserve-self-signed-certs\""
Jul 07 19:40:40.205595 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.205575    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"]
Jul 07 19:40:40.218087 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.218059    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-tmp-dir\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.218200 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.218100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-home\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.218200 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.218115    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-dshm\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.218200 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.218180    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548db76-c1a9-436d-9265-efd86c09b2ba-tls-certs\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.218371 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.218233    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552pz\" (UniqueName: \"kubernetes.io/projected/e548db76-c1a9-436d-9265-efd86c09b2ba-kube-api-access-552pz\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.218371 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.218281    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-model-cache\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.318613 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.318584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552pz\" (UniqueName: \"kubernetes.io/projected/e548db76-c1a9-436d-9265-efd86c09b2ba-kube-api-access-552pz\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.318760 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.318633    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-model-cache\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.318760 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.318656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-tmp-dir\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.318760 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.318679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-home\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.318760 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.318692    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-dshm\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.318760 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.318724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548db76-c1a9-436d-9265-efd86c09b2ba-tls-certs\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.319121 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.319096    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-tmp-dir\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.319182 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.319134    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-home\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.319223 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.319183    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-model-cache\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.321006 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.320987    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-dshm\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.321418 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.321399    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548db76-c1a9-436d-9265-efd86c09b2ba-tls-certs\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.331017 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.330994    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552pz\" (UniqueName: \"kubernetes.io/projected/e548db76-c1a9-436d-9265-efd86c09b2ba-kube-api-access-552pz\") pod \"test-model-548bc4-kserve-9c8968cc-kqs5v\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") " pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.497503 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.497477    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:40.631352 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:40.631323    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"]
Jul 07 19:40:40.634455 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:40:40.634432    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode548db76_c1a9_436d_9265_efd86c09b2ba.slice/crio-15dbe1dd0c0295e81d9c453af18d8c9d221012b65dd4887e11f7ccebf919c997 WatchSource:0}: Error finding container 15dbe1dd0c0295e81d9c453af18d8c9d221012b65dd4887e11f7ccebf919c997: Status 404 returned error can't find the container with id 15dbe1dd0c0295e81d9c453af18d8c9d221012b65dd4887e11f7ccebf919c997
Jul 07 19:40:41.546538 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:41.546498    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v" event={"ID":"e548db76-c1a9-436d-9265-efd86c09b2ba","Type":"ContainerStarted","Data":"ec8e20766230cbfb3c497c448548ab6c613fb26864b808821e3e71aef093ea6c"}
Jul 07 19:40:41.546538 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:41.546544    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v" event={"ID":"e548db76-c1a9-436d-9265-efd86c09b2ba","Type":"ContainerStarted","Data":"15dbe1dd0c0295e81d9c453af18d8c9d221012b65dd4887e11f7ccebf919c997"}
Jul 07 19:40:41.568625 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:41.568572    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v" podStartSLOduration=1.5685562800000001 podStartE2EDuration="1.56855628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:40:41.566254777 +0000 UTC m=+1962.217880710" watchObservedRunningTime="2026-07-07 19:40:41.56855628 +0000 UTC m=+1962.220182212"
Jul 07 19:40:50.498408 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:50.498373    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:50.498763 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:50.498455    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:50.510316 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:50.510298    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:40:50.596687 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:40:50.596657    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:41:14.537400 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:14.537364    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"]
Jul 07 19:41:15.614326 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:15.614293    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"]
Jul 07 19:41:29.556863 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.556820    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t" podUID="736c32f9-4828-4f85-a94b-ff90174429e9" containerName="main" containerID="cri-o://b427cf77c71e0df46e8887991b9e0feb005f0afdee3e5ab7c06ed3f6c81f4fe0" gracePeriod=45
Jul 07 19:41:29.753811 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.753780    2567 generic.go:356] "Generic (PLEG): container finished" podID="736c32f9-4828-4f85-a94b-ff90174429e9" containerID="b427cf77c71e0df46e8887991b9e0feb005f0afdee3e5ab7c06ed3f6c81f4fe0" exitCode=0
Jul 07 19:41:29.753962 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.753825    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t" event={"ID":"736c32f9-4828-4f85-a94b-ff90174429e9","Type":"ContainerDied","Data":"b427cf77c71e0df46e8887991b9e0feb005f0afdee3e5ab7c06ed3f6c81f4fe0"}
Jul 07 19:41:29.793468 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.793446    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:41:29.913905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.913829    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-dshm\") pod \"736c32f9-4828-4f85-a94b-ff90174429e9\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") "
Jul 07 19:41:29.913905 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.913865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-home\") pod \"736c32f9-4828-4f85-a94b-ff90174429e9\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") "
Jul 07 19:41:29.914108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.913941    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-tmp-dir\") pod \"736c32f9-4828-4f85-a94b-ff90174429e9\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") "
Jul 07 19:41:29.914108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.913982    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-model-cache\") pod \"736c32f9-4828-4f85-a94b-ff90174429e9\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") "
Jul 07 19:41:29.914108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.914014    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvg5v\" (UniqueName: \"kubernetes.io/projected/736c32f9-4828-4f85-a94b-ff90174429e9-kube-api-access-lvg5v\") pod \"736c32f9-4828-4f85-a94b-ff90174429e9\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") "
Jul 07 19:41:29.914108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.914073    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736c32f9-4828-4f85-a94b-ff90174429e9-tls-certs\") pod \"736c32f9-4828-4f85-a94b-ff90174429e9\" (UID: \"736c32f9-4828-4f85-a94b-ff90174429e9\") "
Jul 07 19:41:29.914379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.914225    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-home" (OuterVolumeSpecName: "home") pod "736c32f9-4828-4f85-a94b-ff90174429e9" (UID: "736c32f9-4828-4f85-a94b-ff90174429e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:29.914379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.914298    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "736c32f9-4828-4f85-a94b-ff90174429e9" (UID: "736c32f9-4828-4f85-a94b-ff90174429e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:29.914379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.914313    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "736c32f9-4828-4f85-a94b-ff90174429e9" (UID: "736c32f9-4828-4f85-a94b-ff90174429e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:29.914379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.914313    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:29.916231 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.916197    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-dshm" (OuterVolumeSpecName: "dshm") pod "736c32f9-4828-4f85-a94b-ff90174429e9" (UID: "736c32f9-4828-4f85-a94b-ff90174429e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:29.916352 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.916269    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736c32f9-4828-4f85-a94b-ff90174429e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "736c32f9-4828-4f85-a94b-ff90174429e9" (UID: "736c32f9-4828-4f85-a94b-ff90174429e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:41:29.916352 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:29.916315    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736c32f9-4828-4f85-a94b-ff90174429e9-kube-api-access-lvg5v" (OuterVolumeSpecName: "kube-api-access-lvg5v") pod "736c32f9-4828-4f85-a94b-ff90174429e9" (UID: "736c32f9-4828-4f85-a94b-ff90174429e9"). InnerVolumeSpecName "kube-api-access-lvg5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:41:30.015140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.015115    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvg5v\" (UniqueName: \"kubernetes.io/projected/736c32f9-4828-4f85-a94b-ff90174429e9-kube-api-access-lvg5v\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:30.015140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.015136    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736c32f9-4828-4f85-a94b-ff90174429e9-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:30.015140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.015144    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:30.015324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.015151    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:30.015324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.015159    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736c32f9-4828-4f85-a94b-ff90174429e9-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:30.634153 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.634111    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v" podUID="e548db76-c1a9-436d-9265-efd86c09b2ba" containerName="main" containerID="cri-o://ec8e20766230cbfb3c497c448548ab6c613fb26864b808821e3e71aef093ea6c" gracePeriod=45
Jul 07 19:41:30.759452 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.759426    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"
Jul 07 19:41:30.759586 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.759427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t" event={"ID":"736c32f9-4828-4f85-a94b-ff90174429e9","Type":"ContainerDied","Data":"0036e99ced0840dfc97f382321263db3e7e3f219f85e0cb3ee6159029974d330"}
Jul 07 19:41:30.759586 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.759548    2567 scope.go:117] "RemoveContainer" containerID="b427cf77c71e0df46e8887991b9e0feb005f0afdee3e5ab7c06ed3f6c81f4fe0"
Jul 07 19:41:30.761337 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.761313    2567 generic.go:356] "Generic (PLEG): container finished" podID="e548db76-c1a9-436d-9265-efd86c09b2ba" containerID="ec8e20766230cbfb3c497c448548ab6c613fb26864b808821e3e71aef093ea6c" exitCode=0
Jul 07 19:41:30.761461 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.761396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v" event={"ID":"e548db76-c1a9-436d-9265-efd86c09b2ba","Type":"ContainerDied","Data":"ec8e20766230cbfb3c497c448548ab6c613fb26864b808821e3e71aef093ea6c"}
Jul 07 19:41:30.788303 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.788271    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"]
Jul 07 19:41:30.790809 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.790786    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-444f1d/test-model-444f1d-kserve-6b567ffcd7-5f59t"]
Jul 07 19:41:30.892317 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:30.892256    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:41:31.023822 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.023796    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-dshm\") pod \"e548db76-c1a9-436d-9265-efd86c09b2ba\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") "
Jul 07 19:41:31.023983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.023839    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-model-cache\") pod \"e548db76-c1a9-436d-9265-efd86c09b2ba\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") "
Jul 07 19:41:31.023983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.023881    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548db76-c1a9-436d-9265-efd86c09b2ba-tls-certs\") pod \"e548db76-c1a9-436d-9265-efd86c09b2ba\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") "
Jul 07 19:41:31.023983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.023972    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-552pz\" (UniqueName: \"kubernetes.io/projected/e548db76-c1a9-436d-9265-efd86c09b2ba-kube-api-access-552pz\") pod \"e548db76-c1a9-436d-9265-efd86c09b2ba\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") "
Jul 07 19:41:31.024108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.024005    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-tmp-dir\") pod \"e548db76-c1a9-436d-9265-efd86c09b2ba\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") "
Jul 07 19:41:31.024108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.024036    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-home\") pod \"e548db76-c1a9-436d-9265-efd86c09b2ba\" (UID: \"e548db76-c1a9-436d-9265-efd86c09b2ba\") "
Jul 07 19:41:31.024209 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.024158    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "e548db76-c1a9-436d-9265-efd86c09b2ba" (UID: "e548db76-c1a9-436d-9265-efd86c09b2ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:31.024296 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.024274    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e548db76-c1a9-436d-9265-efd86c09b2ba" (UID: "e548db76-c1a9-436d-9265-efd86c09b2ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:31.024362 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.024279    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:31.024416 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.024388    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-home" (OuterVolumeSpecName: "home") pod "e548db76-c1a9-436d-9265-efd86c09b2ba" (UID: "e548db76-c1a9-436d-9265-efd86c09b2ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:31.026093 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.026065    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e548db76-c1a9-436d-9265-efd86c09b2ba-kube-api-access-552pz" (OuterVolumeSpecName: "kube-api-access-552pz") pod "e548db76-c1a9-436d-9265-efd86c09b2ba" (UID: "e548db76-c1a9-436d-9265-efd86c09b2ba"). InnerVolumeSpecName "kube-api-access-552pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:41:31.026197 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.026103    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-dshm" (OuterVolumeSpecName: "dshm") pod "e548db76-c1a9-436d-9265-efd86c09b2ba" (UID: "e548db76-c1a9-436d-9265-efd86c09b2ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:41:31.026197 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.026106    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e548db76-c1a9-436d-9265-efd86c09b2ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e548db76-c1a9-436d-9265-efd86c09b2ba" (UID: "e548db76-c1a9-436d-9265-efd86c09b2ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:41:31.125274 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.125250    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548db76-c1a9-436d-9265-efd86c09b2ba-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:31.125274 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.125271    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552pz\" (UniqueName: \"kubernetes.io/projected/e548db76-c1a9-436d-9265-efd86c09b2ba-kube-api-access-552pz\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:31.125431 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.125281    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:31.125431 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.125289    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:31.125431 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.125298    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548db76-c1a9-436d-9265-efd86c09b2ba-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:41:31.767771 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.767731    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v" event={"ID":"e548db76-c1a9-436d-9265-efd86c09b2ba","Type":"ContainerDied","Data":"15dbe1dd0c0295e81d9c453af18d8c9d221012b65dd4887e11f7ccebf919c997"}
Jul 07 19:41:31.767771 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.767763    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"
Jul 07 19:41:31.768287 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.767789    2567 scope.go:117] "RemoveContainer" containerID="ec8e20766230cbfb3c497c448548ab6c613fb26864b808821e3e71aef093ea6c"
Jul 07 19:41:31.821549 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.821520    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"]
Jul 07 19:41:31.850294 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.850267    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736c32f9-4828-4f85-a94b-ff90174429e9" path="/var/lib/kubelet/pods/736c32f9-4828-4f85-a94b-ff90174429e9/volumes"
Jul 07 19:41:31.852702 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:31.852683    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-548bc4/test-model-548bc4-kserve-9c8968cc-kqs5v"]
Jul 07 19:41:33.849673 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:41:33.849637    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e548db76-c1a9-436d-9265-efd86c09b2ba" path="/var/lib/kubelet/pods/e548db76-c1a9-436d-9265-efd86c09b2ba/volumes"
Jul 07 19:42:59.829345 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:42:59.829313    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:42:59.837166 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:42:59.837147    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:43:25.246006 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.245975    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"]
Jul 07 19:43:25.246459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.246288    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736c32f9-4828-4f85-a94b-ff90174429e9" containerName="main"
Jul 07 19:43:25.246459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.246299    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="736c32f9-4828-4f85-a94b-ff90174429e9" containerName="main"
Jul 07 19:43:25.246459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.246313    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e548db76-c1a9-436d-9265-efd86c09b2ba" containerName="main"
Jul 07 19:43:25.246459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.246318    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e548db76-c1a9-436d-9265-efd86c09b2ba" containerName="main"
Jul 07 19:43:25.246459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.246371    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e548db76-c1a9-436d-9265-efd86c09b2ba" containerName="main"
Jul 07 19:43:25.246459 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.246378    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="736c32f9-4828-4f85-a94b-ff90174429e9" containerName="main"
Jul 07 19:43:25.249239 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.249221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.252065 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.252038    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"openshift-service-ca.crt\""
Jul 07 19:43:25.252818 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.252799    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"default-dockercfg-cdqvr\""
Jul 07 19:43:25.252940 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.252801    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"kube-root-ca.crt\""
Jul 07 19:43:25.253000 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.252937    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"auth-test-model-778336-kserve-self-signed-certs\""
Jul 07 19:43:25.259341 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.259316    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"]
Jul 07 19:43:25.320187 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.320151    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-tmp-dir\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.320342 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.320198    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9xgv\" (UniqueName: \"kubernetes.io/projected/d063a7b8-d340-400e-9cba-8ba340793b1a-kube-api-access-m9xgv\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.320342 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.320225    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d063a7b8-d340-400e-9cba-8ba340793b1a-tls-certs\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.320342 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.320248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-home\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.320342 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.320315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-dshm\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.320513 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.320379    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-model-cache\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.420742 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.420705    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d063a7b8-d340-400e-9cba-8ba340793b1a-tls-certs\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.420742 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.420743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-home\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.420987 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.420779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-dshm\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.420987 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.420811    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-model-cache\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.420987 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.420839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-tmp-dir\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.420987 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.420855    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9xgv\" (UniqueName: \"kubernetes.io/projected/d063a7b8-d340-400e-9cba-8ba340793b1a-kube-api-access-m9xgv\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.421252 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.421229    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-home\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.421349 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.421311    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-tmp-dir\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.421403 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.421354    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-model-cache\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.423246 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.423229    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-dshm\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.423309 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.423237    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d063a7b8-d340-400e-9cba-8ba340793b1a-tls-certs\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.430158 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.430139    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9xgv\" (UniqueName: \"kubernetes.io/projected/d063a7b8-d340-400e-9cba-8ba340793b1a-kube-api-access-m9xgv\") pod \"auth-test-model-778336-kserve-644758d96c-p7jpx\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") " pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.558428 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.558346    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:25.692038 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:25.692014    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"]
Jul 07 19:43:25.694250 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:43:25.694223    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd063a7b8_d340_400e_9cba_8ba340793b1a.slice/crio-9b70d22bf1e59c33ae7175e23240495f06d26fd219894b5f65ca8cb05c27a22d WatchSource:0}: Error finding container 9b70d22bf1e59c33ae7175e23240495f06d26fd219894b5f65ca8cb05c27a22d: Status 404 returned error can't find the container with id 9b70d22bf1e59c33ae7175e23240495f06d26fd219894b5f65ca8cb05c27a22d
Jul 07 19:43:26.225379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:26.225344    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx" event={"ID":"d063a7b8-d340-400e-9cba-8ba340793b1a","Type":"ContainerStarted","Data":"8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a"}
Jul 07 19:43:26.225379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:26.225383    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx" event={"ID":"d063a7b8-d340-400e-9cba-8ba340793b1a","Type":"ContainerStarted","Data":"9b70d22bf1e59c33ae7175e23240495f06d26fd219894b5f65ca8cb05c27a22d"}
Jul 07 19:43:26.250790 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:26.250740    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx" podStartSLOduration=1.250725338 podStartE2EDuration="1.250725338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:43:26.24975351 +0000 UTC m=+2126.901379443" watchObservedRunningTime="2026-07-07 19:43:26.250725338 +0000 UTC m=+2126.902351285"
Jul 07 19:43:35.559308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:35.559270    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:35.559308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:35.559312    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:35.571863 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:35.571842    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:36.278825 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:36.278793    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:43:59.901801 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.901772    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"]
Jul 07 19:43:59.904907 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.904889    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:43:59.916162 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.916141    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"default-dockercfg-hckbh\""
Jul 07 19:43:59.917518 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.917499    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"openshift-service-ca.crt\""
Jul 07 19:43:59.917744 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.917730    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"auth-test-model-fe4a68-kserve-self-signed-certs\""
Jul 07 19:43:59.923037 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.923019    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"kube-root-ca.crt\""
Jul 07 19:43:59.931776 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:43:59.931754    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"]
Jul 07 19:44:00.005615 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.005588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-home\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.005615 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.005621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlbz\" (UniqueName: \"kubernetes.io/projected/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-kube-api-access-wzlbz\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.005819 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.005673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tmp-dir\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.005819 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.005701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tls-certs\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.005819 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.005723    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-dshm\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.005819 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.005762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-model-cache\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106134 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-dshm\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-model-cache\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106187    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-home\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlbz\" (UniqueName: \"kubernetes.io/projected/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-kube-api-access-wzlbz\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106227    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tmp-dir\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106308 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106251    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tls-certs\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106653 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106625    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-model-cache\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106653 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106643    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tmp-dir\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.106951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.106675    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-home\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.108559 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.108534    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-dshm\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.108671 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.108655    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tls-certs\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.115491 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.115467    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlbz\" (UniqueName: \"kubernetes.io/projected/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-kube-api-access-wzlbz\") pod \"auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") " pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.212812 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.212744    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:00.354260 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.354236    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"]
Jul 07 19:44:00.356550 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:44:00.356514    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c9b28a2_f98f_45d5_bf0a_bc00f54f1002.slice/crio-81ee67fceff3b4e29b54470c185902ce02e6dda5e641ca4be00e3e262665d939 WatchSource:0}: Error finding container 81ee67fceff3b4e29b54470c185902ce02e6dda5e641ca4be00e3e262665d939: Status 404 returned error can't find the container with id 81ee67fceff3b4e29b54470c185902ce02e6dda5e641ca4be00e3e262665d939
Jul 07 19:44:00.365700 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:00.365672    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm" event={"ID":"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002","Type":"ContainerStarted","Data":"81ee67fceff3b4e29b54470c185902ce02e6dda5e641ca4be00e3e262665d939"}
Jul 07 19:44:01.375939 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:01.375870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm" event={"ID":"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002","Type":"ContainerStarted","Data":"81b15b367fe32ecd6c79a3a83965dc7d9971dfdfcb07bc724e8cc734f8a460a9"}
Jul 07 19:44:10.213040 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:10.212994    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:10.213446 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:10.213065    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:10.227184 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:10.227161    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:44:10.262133 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:10.262091    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm" podStartSLOduration=11.262076862 podStartE2EDuration="11.262076862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:44:01.400169024 +0000 UTC m=+2162.051794956" watchObservedRunningTime="2026-07-07 19:44:10.262076862 +0000 UTC m=+2170.913702794"
Jul 07 19:44:10.425009 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:44:10.424981    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:46:06.688108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:06.688073    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"]
Jul 07 19:46:07.762194 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:07.762159    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"]
Jul 07 19:46:21.707484 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.707427    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx" podUID="d063a7b8-d340-400e-9cba-8ba340793b1a" containerName="main" containerID="cri-o://8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a" gracePeriod=45
Jul 07 19:46:21.954304 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.954275    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:46:21.963474 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.963418    2567 generic.go:356] "Generic (PLEG): container finished" podID="d063a7b8-d340-400e-9cba-8ba340793b1a" containerID="8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a" exitCode=0
Jul 07 19:46:21.963474 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.963454    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"
Jul 07 19:46:21.963474 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.963470    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx" event={"ID":"d063a7b8-d340-400e-9cba-8ba340793b1a","Type":"ContainerDied","Data":"8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a"}
Jul 07 19:46:21.963703 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.963511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx" event={"ID":"d063a7b8-d340-400e-9cba-8ba340793b1a","Type":"ContainerDied","Data":"9b70d22bf1e59c33ae7175e23240495f06d26fd219894b5f65ca8cb05c27a22d"}
Jul 07 19:46:21.963703 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.963537    2567 scope.go:117] "RemoveContainer" containerID="8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a"
Jul 07 19:46:21.971885 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.971869    2567 scope.go:117] "RemoveContainer" containerID="8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a"
Jul 07 19:46:21.972174 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:46:21.972156    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a\": container with ID starting with 8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a not found: ID does not exist" containerID="8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a"
Jul 07 19:46:21.972240 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:21.972182    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a"} err="failed to get container status \"8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a\": rpc error: code = NotFound desc = could not find container \"8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a\": container with ID starting with 8794c39f4ad1f732e7fd6531cc7dcd89e53cb405d3ebeeb1117e495c02c0b52a not found: ID does not exist"
Jul 07 19:46:22.078952 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.078909    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d063a7b8-d340-400e-9cba-8ba340793b1a-tls-certs\") pod \"d063a7b8-d340-400e-9cba-8ba340793b1a\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") "
Jul 07 19:46:22.078952 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.078957    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-dshm\") pod \"d063a7b8-d340-400e-9cba-8ba340793b1a\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") "
Jul 07 19:46:22.079176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.078989    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-home\") pod \"d063a7b8-d340-400e-9cba-8ba340793b1a\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") "
Jul 07 19:46:22.079176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.079009    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-model-cache\") pod \"d063a7b8-d340-400e-9cba-8ba340793b1a\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") "
Jul 07 19:46:22.079176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.079061    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-tmp-dir\") pod \"d063a7b8-d340-400e-9cba-8ba340793b1a\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") "
Jul 07 19:46:22.079176 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.079077    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9xgv\" (UniqueName: \"kubernetes.io/projected/d063a7b8-d340-400e-9cba-8ba340793b1a-kube-api-access-m9xgv\") pod \"d063a7b8-d340-400e-9cba-8ba340793b1a\" (UID: \"d063a7b8-d340-400e-9cba-8ba340793b1a\") "
Jul 07 19:46:22.079361 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.079264    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-home" (OuterVolumeSpecName: "home") pod "d063a7b8-d340-400e-9cba-8ba340793b1a" (UID: "d063a7b8-d340-400e-9cba-8ba340793b1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:22.079402 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.079352    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d063a7b8-d340-400e-9cba-8ba340793b1a" (UID: "d063a7b8-d340-400e-9cba-8ba340793b1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:22.079402 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.079366    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d063a7b8-d340-400e-9cba-8ba340793b1a" (UID: "d063a7b8-d340-400e-9cba-8ba340793b1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:22.081199 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.081171    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d063a7b8-d340-400e-9cba-8ba340793b1a-kube-api-access-m9xgv" (OuterVolumeSpecName: "kube-api-access-m9xgv") pod "d063a7b8-d340-400e-9cba-8ba340793b1a" (UID: "d063a7b8-d340-400e-9cba-8ba340793b1a"). InnerVolumeSpecName "kube-api-access-m9xgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:46:22.081464 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.081442    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d063a7b8-d340-400e-9cba-8ba340793b1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d063a7b8-d340-400e-9cba-8ba340793b1a" (UID: "d063a7b8-d340-400e-9cba-8ba340793b1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:46:22.081464 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.081449    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-dshm" (OuterVolumeSpecName: "dshm") pod "d063a7b8-d340-400e-9cba-8ba340793b1a" (UID: "d063a7b8-d340-400e-9cba-8ba340793b1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:22.180297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.180267    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:22.180297 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.180291    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:22.180453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.180303    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:22.180453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.180316    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d063a7b8-d340-400e-9cba-8ba340793b1a-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:22.180453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.180328    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9xgv\" (UniqueName: \"kubernetes.io/projected/d063a7b8-d340-400e-9cba-8ba340793b1a-kube-api-access-m9xgv\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:22.180453 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.180339    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d063a7b8-d340-400e-9cba-8ba340793b1a-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:22.287946 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.287900    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"]
Jul 07 19:46:22.290943 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.290905    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-778336/auth-test-model-778336-kserve-644758d96c-p7jpx"]
Jul 07 19:46:22.781865 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.781830    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm" podUID="4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" containerName="main" containerID="cri-o://81b15b367fe32ecd6c79a3a83965dc7d9971dfdfcb07bc724e8cc734f8a460a9" gracePeriod=45
Jul 07 19:46:22.970444 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.970415    2567 generic.go:356] "Generic (PLEG): container finished" podID="4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" containerID="81b15b367fe32ecd6c79a3a83965dc7d9971dfdfcb07bc724e8cc734f8a460a9" exitCode=0
Jul 07 19:46:22.970586 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:22.970493    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm" event={"ID":"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002","Type":"ContainerDied","Data":"81b15b367fe32ecd6c79a3a83965dc7d9971dfdfcb07bc724e8cc734f8a460a9"}
Jul 07 19:46:23.030674 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.030652    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:46:23.191211 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191124    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-model-cache\") pod \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") "
Jul 07 19:46:23.191211 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191171    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-dshm\") pod \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") "
Jul 07 19:46:23.191211 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191199    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tmp-dir\") pod \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") "
Jul 07 19:46:23.191211 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191219    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzlbz\" (UniqueName: \"kubernetes.io/projected/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-kube-api-access-wzlbz\") pod \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") "
Jul 07 19:46:23.191532 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191234    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-home\") pod \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") "
Jul 07 19:46:23.191532 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191266    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tls-certs\") pod \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\" (UID: \"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002\") "
Jul 07 19:46:23.191532 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191413    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" (UID: "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:23.191532 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191494    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-home" (OuterVolumeSpecName: "home") pod "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" (UID: "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:23.191532 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191504    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" (UID: "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:23.191746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191592    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:23.191746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191612    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:23.191746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.191654    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:23.193523 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.193498    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-kube-api-access-wzlbz" (OuterVolumeSpecName: "kube-api-access-wzlbz") pod "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" (UID: "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002"). InnerVolumeSpecName "kube-api-access-wzlbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:46:23.193596 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.193522    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-dshm" (OuterVolumeSpecName: "dshm") pod "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" (UID: "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:46:23.193631 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.193597    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" (UID: "4c9b28a2-f98f-45d5-bf0a-bc00f54f1002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:46:23.292645 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.292617    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:23.292645 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.292642    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzlbz\" (UniqueName: \"kubernetes.io/projected/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-kube-api-access-wzlbz\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:23.292798 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.292652    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:46:23.850339 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.850305    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d063a7b8-d340-400e-9cba-8ba340793b1a" path="/var/lib/kubelet/pods/d063a7b8-d340-400e-9cba-8ba340793b1a/volumes"
Jul 07 19:46:23.977096 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.977069    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"
Jul 07 19:46:23.977258 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.977095    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm" event={"ID":"4c9b28a2-f98f-45d5-bf0a-bc00f54f1002","Type":"ContainerDied","Data":"81ee67fceff3b4e29b54470c185902ce02e6dda5e641ca4be00e3e262665d939"}
Jul 07 19:46:23.977258 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.977140    2567 scope.go:117] "RemoveContainer" containerID="81b15b367fe32ecd6c79a3a83965dc7d9971dfdfcb07bc724e8cc734f8a460a9"
Jul 07 19:46:23.994443 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:23.994421    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"]
Jul 07 19:46:24.000788 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:24.000765    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/auth-test-model-fe4a68-kserve-557c77cfbc-xzqgm"]
Jul 07 19:46:25.850291 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:46:25.850258    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" path="/var/lib/kubelet/pods/4c9b28a2-f98f-45d5-bf0a-bc00f54f1002/volumes"
Jul 07 19:47:01.566520 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566481    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"]
Jul 07 19:47:01.567012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566812    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" containerName="main"
Jul 07 19:47:01.567012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566825    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" containerName="main"
Jul 07 19:47:01.567012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566835    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d063a7b8-d340-400e-9cba-8ba340793b1a" containerName="main"
Jul 07 19:47:01.567012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566840    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d063a7b8-d340-400e-9cba-8ba340793b1a" containerName="main"
Jul 07 19:47:01.567012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566904    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d063a7b8-d340-400e-9cba-8ba340793b1a" containerName="main"
Jul 07 19:47:01.567012 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.566938    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9b28a2-f98f-45d5-bf0a-bc00f54f1002" containerName="main"
Jul 07 19:47:01.571174 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.571158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.575968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.575944    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"kube-root-ca.crt\""
Jul 07 19:47:01.576091 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.575970    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"sub-test-model-fe4a68-kserve-self-signed-certs\""
Jul 07 19:47:01.576091 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.576003    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"openshift-service-ca.crt\""
Jul 07 19:47:01.576091 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.575970    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"default-dockercfg-hckbh\""
Jul 07 19:47:01.586660 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.586638    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"]
Jul 07 19:47:01.696580 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.696542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-dshm\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.696746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.696611    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tmp-dir\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.696746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.696666    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-home\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.696746 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.696697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tls-certs\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.696855 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.696752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9sj\" (UniqueName: \"kubernetes.io/projected/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-kube-api-access-zg9sj\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.696855 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.696787    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-model-cache\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797325 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-dshm\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797341    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tmp-dir\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-home\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tls-certs\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797396    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9sj\" (UniqueName: \"kubernetes.io/projected/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-kube-api-access-zg9sj\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797501 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-model-cache\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797744 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797718    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tmp-dir\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797803 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797763    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-home\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.797864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.797847    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-model-cache\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.799728 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.799692    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-dshm\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.800054 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.800038    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tls-certs\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.808959 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.808932    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9sj\" (UniqueName: \"kubernetes.io/projected/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-kube-api-access-zg9sj\") pod \"sub-test-model-fe4a68-kserve-694795c5dc-x8pbj\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") " pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:01.880388 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:01.880311    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:02.013034 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:02.013009    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"]
Jul 07 19:47:02.014764 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:47:02.014731    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f6de860_a370_4974_9eb6_4e8c12a7d6c3.slice/crio-c56840c1732b03be9100fbc565b9b12618375cb47980559afecc0d62f064988e WatchSource:0}: Error finding container c56840c1732b03be9100fbc565b9b12618375cb47980559afecc0d62f064988e: Status 404 returned error can't find the container with id c56840c1732b03be9100fbc565b9b12618375cb47980559afecc0d62f064988e
Jul 07 19:47:02.135189 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:02.135114    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj" event={"ID":"3f6de860-a370-4974-9eb6-4e8c12a7d6c3","Type":"ContainerStarted","Data":"3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468"}
Jul 07 19:47:02.135189 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:02.135157    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj" event={"ID":"3f6de860-a370-4974-9eb6-4e8c12a7d6c3","Type":"ContainerStarted","Data":"c56840c1732b03be9100fbc565b9b12618375cb47980559afecc0d62f064988e"}
Jul 07 19:47:11.880735 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:11.880706    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:11.881203 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:11.880749    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:11.893375 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:11.893354    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:11.915806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:11.915767    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj" podStartSLOduration=10.915754957 podStartE2EDuration="10.915754957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:47:02.16974286 +0000 UTC m=+2342.821368793" watchObservedRunningTime="2026-07-07 19:47:11.915754957 +0000 UTC m=+2352.567380890"
Jul 07 19:47:12.187623 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:12.187553    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:47:55.558400 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:55.558361    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"]
Jul 07 19:47:59.855363 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:59.855336    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:47:59.860964 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:47:59.860944    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47hd8_9b413681-a928-4b3c-a6a5-f14435739728/ovn-acl-logging/0.log"
Jul 07 19:48:04.675083 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.675045    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"]
Jul 07 19:48:04.678148 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.678131    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.681795 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.681773    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"kube-root-ca.crt\""
Jul 07 19:48:04.681951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.681774    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"default-dockercfg-cdqvr\""
Jul 07 19:48:04.681951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.681774    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"rate-test-model-778336-kserve-self-signed-certs\""
Jul 07 19:48:04.681951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.681822    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-778336\"/\"openshift-service-ca.crt\""
Jul 07 19:48:04.691849 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.691828    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"]
Jul 07 19:48:04.744890 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.744860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-dshm\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.745043 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.744898    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-tmp-dir\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.745043 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.744948    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-home\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.745116 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.745046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-model-cache\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.745116 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.745078    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgr6q\" (UniqueName: \"kubernetes.io/projected/299f248b-9b51-4440-875a-c8e1ab8098a6-kube-api-access-vgr6q\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.745184 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.745118    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f248b-9b51-4440-875a-c8e1ab8098a6-tls-certs\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845450 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-dshm\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845450 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845452    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-tmp-dir\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845660 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-home\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845660 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845502    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-model-cache\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845660 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgr6q\" (UniqueName: \"kubernetes.io/projected/299f248b-9b51-4440-875a-c8e1ab8098a6-kube-api-access-vgr6q\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845660 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845561    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f248b-9b51-4440-875a-c8e1ab8098a6-tls-certs\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845903 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845870    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-tmp-dir\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.845978 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845938    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-home\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.846018 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.845985    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-model-cache\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.847650 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.847627    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-dshm\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.847912 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.847894    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f248b-9b51-4440-875a-c8e1ab8098a6-tls-certs\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.857471 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.857442    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgr6q\" (UniqueName: \"kubernetes.io/projected/299f248b-9b51-4440-875a-c8e1ab8098a6-kube-api-access-vgr6q\") pod \"rate-test-model-778336-kserve-69db65b7bc-5qcs9\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") " pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:04.986954 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:04.986911    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:05.118638 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:05.118561    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"]
Jul 07 19:48:05.120878 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:48:05.120852    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod299f248b_9b51_4440_875a_c8e1ab8098a6.slice/crio-df6bfcab3216a6c53401224886483fc5fc04caac1c504cbbd98579e29e366927 WatchSource:0}: Error finding container df6bfcab3216a6c53401224886483fc5fc04caac1c504cbbd98579e29e366927: Status 404 returned error can't find the container with id df6bfcab3216a6c53401224886483fc5fc04caac1c504cbbd98579e29e366927
Jul 07 19:48:05.388239 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:05.388151    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9" event={"ID":"299f248b-9b51-4440-875a-c8e1ab8098a6","Type":"ContainerStarted","Data":"74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19"}
Jul 07 19:48:05.388239 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:05.388193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9" event={"ID":"299f248b-9b51-4440-875a-c8e1ab8098a6","Type":"ContainerStarted","Data":"df6bfcab3216a6c53401224886483fc5fc04caac1c504cbbd98579e29e366927"}
Jul 07 19:48:05.414443 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:05.414394    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9" podStartSLOduration=1.414379003 podStartE2EDuration="1.414379003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:48:05.412741299 +0000 UTC m=+2406.064367232" watchObservedRunningTime="2026-07-07 19:48:05.414379003 +0000 UTC m=+2406.066004936"
Jul 07 19:48:10.578452 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.578414    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj" podUID="3f6de860-a370-4974-9eb6-4e8c12a7d6c3" containerName="main" containerID="cri-o://3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468" gracePeriod=45
Jul 07 19:48:10.822271 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.822243    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:48:10.885237 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885172    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tmp-dir\") pod \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") "
Jul 07 19:48:10.885237 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885218    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-model-cache\") pod \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") "
Jul 07 19:48:10.885435 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885249    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tls-certs\") pod \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") "
Jul 07 19:48:10.885435 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885278    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg9sj\" (UniqueName: \"kubernetes.io/projected/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-kube-api-access-zg9sj\") pod \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") "
Jul 07 19:48:10.885435 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885313    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-home\") pod \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") "
Jul 07 19:48:10.885591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885463    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-dshm\") pod \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\" (UID: \"3f6de860-a370-4974-9eb6-4e8c12a7d6c3\") "
Jul 07 19:48:10.885591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885479    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f6de860-a370-4974-9eb6-4e8c12a7d6c3" (UID: "3f6de860-a370-4974-9eb6-4e8c12a7d6c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:48:10.885591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885534    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-home" (OuterVolumeSpecName: "home") pod "3f6de860-a370-4974-9eb6-4e8c12a7d6c3" (UID: "3f6de860-a370-4974-9eb6-4e8c12a7d6c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:48:10.885591 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885550    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f6de860-a370-4974-9eb6-4e8c12a7d6c3" (UID: "3f6de860-a370-4974-9eb6-4e8c12a7d6c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:48:10.885767 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885745    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:48:10.885806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885771    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:48:10.885806 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.885780    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:48:10.887428 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.887405    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-dshm" (OuterVolumeSpecName: "dshm") pod "3f6de860-a370-4974-9eb6-4e8c12a7d6c3" (UID: "3f6de860-a370-4974-9eb6-4e8c12a7d6c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:48:10.887866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.887849    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f6de860-a370-4974-9eb6-4e8c12a7d6c3" (UID: "3f6de860-a370-4974-9eb6-4e8c12a7d6c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:48:10.887866 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.887855    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-kube-api-access-zg9sj" (OuterVolumeSpecName: "kube-api-access-zg9sj") pod "3f6de860-a370-4974-9eb6-4e8c12a7d6c3" (UID: "3f6de860-a370-4974-9eb6-4e8c12a7d6c3"). InnerVolumeSpecName "kube-api-access-zg9sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:48:10.986467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.986435    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:48:10.986467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.986459    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:48:10.986467 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:10.986469    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg9sj\" (UniqueName: \"kubernetes.io/projected/3f6de860-a370-4974-9eb6-4e8c12a7d6c3-kube-api-access-zg9sj\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:48:11.416183 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.416152    2567 generic.go:356] "Generic (PLEG): container finished" podID="3f6de860-a370-4974-9eb6-4e8c12a7d6c3" containerID="3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468" exitCode=0
Jul 07 19:48:11.416323 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.416208    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"
Jul 07 19:48:11.416323 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.416238    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj" event={"ID":"3f6de860-a370-4974-9eb6-4e8c12a7d6c3","Type":"ContainerDied","Data":"3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468"}
Jul 07 19:48:11.416323 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.416282    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj" event={"ID":"3f6de860-a370-4974-9eb6-4e8c12a7d6c3","Type":"ContainerDied","Data":"c56840c1732b03be9100fbc565b9b12618375cb47980559afecc0d62f064988e"}
Jul 07 19:48:11.416323 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.416304    2567 scope.go:117] "RemoveContainer" containerID="3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468"
Jul 07 19:48:11.424489 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.424466    2567 scope.go:117] "RemoveContainer" containerID="3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468"
Jul 07 19:48:11.424720 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:48:11.424703    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468\": container with ID starting with 3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468 not found: ID does not exist" containerID="3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468"
Jul 07 19:48:11.424771 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.424727    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468"} err="failed to get container status \"3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468\": rpc error: code = NotFound desc = could not find container \"3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468\": container with ID starting with 3754da2572b0fb9c97e731d3035095774523f5089cebd2211126e7e1e248a468 not found: ID does not exist"
Jul 07 19:48:11.439807 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.439781    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"]
Jul 07 19:48:11.448166 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.448137    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/sub-test-model-fe4a68-kserve-694795c5dc-x8pbj"]
Jul 07 19:48:11.850563 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:11.850527    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f6de860-a370-4974-9eb6-4e8c12a7d6c3" path="/var/lib/kubelet/pods/3f6de860-a370-4974-9eb6-4e8c12a7d6c3/volumes"
Jul 07 19:48:14.987692 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:14.987656    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:14.988070 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:14.987956    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:15.000389 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:15.000366    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:15.443913 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:15.443840    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:48:23.283983 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.283887    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"]
Jul 07 19:48:23.284335 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.284195    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f6de860-a370-4974-9eb6-4e8c12a7d6c3" containerName="main"
Jul 07 19:48:23.284335 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.284208    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f6de860-a370-4974-9eb6-4e8c12a7d6c3" containerName="main"
Jul 07 19:48:23.284335 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.284284    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f6de860-a370-4974-9eb6-4e8c12a7d6c3" containerName="main"
Jul 07 19:48:23.288458 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.288439    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.296144 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.296123    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"default-dockercfg-hckbh\""
Jul 07 19:48:23.296144 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.296144    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"openshift-service-ca.crt\""
Jul 07 19:48:23.296324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.296169    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"kube-root-ca.crt\""
Jul 07 19:48:23.296324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.296121    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fe4a68\"/\"rate-test-model-fe4a68-kserve-self-signed-certs\""
Jul 07 19:48:23.308976 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.308949    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"]
Jul 07 19:48:23.381411 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.381374    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-dshm\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.381560 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.381422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-home\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.381560 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.381468    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-model-cache\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.381560 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.381516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-tmp-dir\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.381734 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.381607    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6hlg\" (UniqueName: \"kubernetes.io/projected/f725e886-287b-40c2-9627-3b0142575695-kube-api-access-l6hlg\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.381734 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.381653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f725e886-287b-40c2-9627-3b0142575695-tls-certs\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.482763 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.482730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f725e886-287b-40c2-9627-3b0142575695-tls-certs\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.482968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.482782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-dshm\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.482968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.482810    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-home\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.482968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.482832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-model-cache\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.482968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.482869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-tmp-dir\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.482968 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.482955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6hlg\" (UniqueName: \"kubernetes.io/projected/f725e886-287b-40c2-9627-3b0142575695-kube-api-access-l6hlg\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.483379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.483342    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-tmp-dir\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.483379 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.483356    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-model-cache\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.483529 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.483400    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-home\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.485105 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.485086    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-dshm\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.485490 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.485470    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f725e886-287b-40c2-9627-3b0142575695-tls-certs\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.492291 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.492269    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6hlg\" (UniqueName: \"kubernetes.io/projected/f725e886-287b-40c2-9627-3b0142575695-kube-api-access-l6hlg\") pod \"rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") " pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.597389 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.597315    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:23.729737 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:23.729707    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"]
Jul 07 19:48:23.731696 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:48:23.731662    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf725e886_287b_40c2_9627_3b0142575695.slice/crio-2bd6ca60e1e64f32772bc67b2d1fca8cfe7a4dc54da009146dc30f623cd4813c WatchSource:0}: Error finding container 2bd6ca60e1e64f32772bc67b2d1fca8cfe7a4dc54da009146dc30f623cd4813c: Status 404 returned error can't find the container with id 2bd6ca60e1e64f32772bc67b2d1fca8cfe7a4dc54da009146dc30f623cd4813c
Jul 07 19:48:24.468610 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:24.468571    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq" event={"ID":"f725e886-287b-40c2-9627-3b0142575695","Type":"ContainerStarted","Data":"c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72"}
Jul 07 19:48:24.468610 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:24.468615    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq" event={"ID":"f725e886-287b-40c2-9627-3b0142575695","Type":"ContainerStarted","Data":"2bd6ca60e1e64f32772bc67b2d1fca8cfe7a4dc54da009146dc30f623cd4813c"}
Jul 07 19:48:24.495361 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:24.495317    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq" podStartSLOduration=1.495301763 podStartE2EDuration="1.495301763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:48:24.494581369 +0000 UTC m=+2425.146207301" watchObservedRunningTime="2026-07-07 19:48:24.495301763 +0000 UTC m=+2425.146927696"
Jul 07 19:48:33.598338 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:33.598306    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:33.598787 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:33.598353    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:33.610669 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:33.610640    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:48:34.520295 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:48:34.520266    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:49:14.233503 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:14.233460    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"]
Jul 07 19:49:15.086102 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:15.086069    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"]
Jul 07 19:49:22.491078 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.491040    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7392-140448517755792"]
Jul 07 19:49:22.494324 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.494306    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:22.497194 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.497168    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6rqr\""
Jul 07 19:49:22.503254 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.503229    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7392-140448517755792"]
Jul 07 19:49:22.575232 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.575199    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4ww\" (UniqueName: \"kubernetes.io/projected/c481ad54-13f3-4d6d-bcf5-256425770d55-kube-api-access-7g4ww\") pod \"test-curl-7392-140448517755792\" (UID: \"c481ad54-13f3-4d6d-bcf5-256425770d55\") " pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:22.676011 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.675974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4ww\" (UniqueName: \"kubernetes.io/projected/c481ad54-13f3-4d6d-bcf5-256425770d55-kube-api-access-7g4ww\") pod \"test-curl-7392-140448517755792\" (UID: \"c481ad54-13f3-4d6d-bcf5-256425770d55\") " pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:22.686590 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.686555    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4ww\" (UniqueName: \"kubernetes.io/projected/c481ad54-13f3-4d6d-bcf5-256425770d55-kube-api-access-7g4ww\") pod \"test-curl-7392-140448517755792\" (UID: \"c481ad54-13f3-4d6d-bcf5-256425770d55\") " pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:22.804046 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.803966    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:22.938407 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.938328    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7392-140448517755792"]
Jul 07 19:49:22.941252 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:49:22.941223    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc481ad54_13f3_4d6d_bcf5_256425770d55.slice/crio-8ae33ec7b66e1af6faa0d581d738a7a9dbdfb24cd4d07f3a5d424a237dc8844c WatchSource:0}: Error finding container 8ae33ec7b66e1af6faa0d581d738a7a9dbdfb24cd4d07f3a5d424a237dc8844c: Status 404 returned error can't find the container with id 8ae33ec7b66e1af6faa0d581d738a7a9dbdfb24cd4d07f3a5d424a237dc8844c
Jul 07 19:49:22.942561 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:22.942541    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 19:49:23.726864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:23.726824    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517755792" event={"ID":"c481ad54-13f3-4d6d-bcf5-256425770d55","Type":"ContainerStarted","Data":"d697dfe8e3417cefd8be3baa045db88bf80b1ac76d30d23ec5c33afdd59b76fe"}
Jul 07 19:49:23.727245 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:23.726881    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517755792" event={"ID":"c481ad54-13f3-4d6d-bcf5-256425770d55","Type":"ContainerStarted","Data":"8ae33ec7b66e1af6faa0d581d738a7a9dbdfb24cd4d07f3a5d424a237dc8844c"}
Jul 07 19:49:23.771579 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:23.771532    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7392-140448517755792" podStartSLOduration=1.074650094 podStartE2EDuration="1.771514805s" totalImagesPullingTime="696.864711ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:49:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:49:23.770677185 +0000 UTC m=+2484.422303121" watchObservedRunningTime="2026-07-07 19:49:23.771514805 +0000 UTC m=+2484.423140738"
Jul 07 19:49:23.949080 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:23.949001    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7392-140448517755792_c481ad54-13f3-4d6d-bcf5-256425770d55/test-curl-7392-140448517755792/0.log"
Jul 07 19:49:24.732411 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:24.732380    2567 generic.go:356] "Generic (PLEG): container finished" podID="c481ad54-13f3-4d6d-bcf5-256425770d55" containerID="d697dfe8e3417cefd8be3baa045db88bf80b1ac76d30d23ec5c33afdd59b76fe" exitCode=0
Jul 07 19:49:24.732804 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:24.732467    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517755792" event={"ID":"c481ad54-13f3-4d6d-bcf5-256425770d55","Type":"ContainerDied","Data":"d697dfe8e3417cefd8be3baa045db88bf80b1ac76d30d23ec5c33afdd59b76fe"}
Jul 07 19:49:25.873007 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:25.872984    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:25.911425 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:25.911389    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7392-140448517755792"]
Jul 07 19:49:25.918084 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:25.918056    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7392-140448517755792"]
Jul 07 19:49:26.007151 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:26.007098    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7g4ww\" (UniqueName: \"kubernetes.io/projected/c481ad54-13f3-4d6d-bcf5-256425770d55-kube-api-access-7g4ww\") pod \"c481ad54-13f3-4d6d-bcf5-256425770d55\" (UID: \"c481ad54-13f3-4d6d-bcf5-256425770d55\") "
Jul 07 19:49:26.009186 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:26.009162    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c481ad54-13f3-4d6d-bcf5-256425770d55-kube-api-access-7g4ww" (OuterVolumeSpecName: "kube-api-access-7g4ww") pod "c481ad54-13f3-4d6d-bcf5-256425770d55" (UID: "c481ad54-13f3-4d6d-bcf5-256425770d55"). InnerVolumeSpecName "kube-api-access-7g4ww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:49:26.107722 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:26.107694    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g4ww\" (UniqueName: \"kubernetes.io/projected/c481ad54-13f3-4d6d-bcf5-256425770d55-kube-api-access-7g4ww\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:26.742018 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:26.741987    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae33ec7b66e1af6faa0d581d738a7a9dbdfb24cd4d07f3a5d424a237dc8844c"
Jul 07 19:49:26.742018 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:26.741991    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:26.757231 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:49:26.757200    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7392-140448517755792\" is forbidden: User \"system:node:ip-10-0-141-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-187.ec2.internal' and this object" podUID="c481ad54-13f3-4d6d-bcf5-256425770d55" pod="opendatahub/test-curl-7392-140448517755792"
Jul 07 19:49:27.850273 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:27.850246    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c481ad54-13f3-4d6d-bcf5-256425770d55" path="/var/lib/kubelet/pods/c481ad54-13f3-4d6d-bcf5-256425770d55/volumes"
Jul 07 19:49:28.503108 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.503077    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:28.503398 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.503384    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c481ad54-13f3-4d6d-bcf5-256425770d55" containerName="test-curl-7392-140448517755792"
Jul 07 19:49:28.503451 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.503400    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c481ad54-13f3-4d6d-bcf5-256425770d55" containerName="test-curl-7392-140448517755792"
Jul 07 19:49:28.503485 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.503475    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c481ad54-13f3-4d6d-bcf5-256425770d55" containerName="test-curl-7392-140448517755792"
Jul 07 19:49:28.506490 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.506474    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:28.509044 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.509025    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-v6rqr\""
Jul 07 19:49:28.514048 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.514029    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:28.628667 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.628642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clw6x\" (UniqueName: \"kubernetes.io/projected/e799d733-4ad2-422d-8298-1845d7b988c1-kube-api-access-clw6x\") pod \"test-curl-7392-140448517756512\" (UID: \"e799d733-4ad2-422d-8298-1845d7b988c1\") " pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:28.729573 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.729546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clw6x\" (UniqueName: \"kubernetes.io/projected/e799d733-4ad2-422d-8298-1845d7b988c1-kube-api-access-clw6x\") pod \"test-curl-7392-140448517756512\" (UID: \"e799d733-4ad2-422d-8298-1845d7b988c1\") " pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:28.739529 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.739511    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clw6x\" (UniqueName: \"kubernetes.io/projected/e799d733-4ad2-422d-8298-1845d7b988c1-kube-api-access-clw6x\") pod \"test-curl-7392-140448517756512\" (UID: \"e799d733-4ad2-422d-8298-1845d7b988c1\") " pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:28.815747 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.815681    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:28.960877 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:28.960853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:28.963106 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:49:28.963079    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode799d733_4ad2_422d_8298_1845d7b988c1.slice/crio-a5f6e6b433125d4c1b0ebc2a840b018f1e5e78056de23cf24e0211583ab3f55f WatchSource:0}: Error finding container a5f6e6b433125d4c1b0ebc2a840b018f1e5e78056de23cf24e0211583ab3f55f: Status 404 returned error can't find the container with id a5f6e6b433125d4c1b0ebc2a840b018f1e5e78056de23cf24e0211583ab3f55f
Jul 07 19:49:29.253133 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.253105    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9" podUID="299f248b-9b51-4440-875a-c8e1ab8098a6" containerName="main" containerID="cri-o://74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19" gracePeriod=45
Jul 07 19:49:29.523351 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.523290    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:49:29.637478 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637436    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-home\") pod \"299f248b-9b51-4440-875a-c8e1ab8098a6\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") "
Jul 07 19:49:29.637644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637493    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f248b-9b51-4440-875a-c8e1ab8098a6-tls-certs\") pod \"299f248b-9b51-4440-875a-c8e1ab8098a6\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") "
Jul 07 19:49:29.637644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637532    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgr6q\" (UniqueName: \"kubernetes.io/projected/299f248b-9b51-4440-875a-c8e1ab8098a6-kube-api-access-vgr6q\") pod \"299f248b-9b51-4440-875a-c8e1ab8098a6\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") "
Jul 07 19:49:29.637644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637560    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-model-cache\") pod \"299f248b-9b51-4440-875a-c8e1ab8098a6\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") "
Jul 07 19:49:29.637644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637593    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-dshm\") pod \"299f248b-9b51-4440-875a-c8e1ab8098a6\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") "
Jul 07 19:49:29.637644 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637631    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-tmp-dir\") pod \"299f248b-9b51-4440-875a-c8e1ab8098a6\" (UID: \"299f248b-9b51-4440-875a-c8e1ab8098a6\") "
Jul 07 19:49:29.637892 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637712    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-home" (OuterVolumeSpecName: "home") pod "299f248b-9b51-4440-875a-c8e1ab8098a6" (UID: "299f248b-9b51-4440-875a-c8e1ab8098a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:29.637892 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637807    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "299f248b-9b51-4440-875a-c8e1ab8098a6" (UID: "299f248b-9b51-4440-875a-c8e1ab8098a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:29.637892 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637886    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:29.638059 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637940    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:29.638059 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.637985    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "299f248b-9b51-4440-875a-c8e1ab8098a6" (UID: "299f248b-9b51-4440-875a-c8e1ab8098a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:29.639614 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.639591    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-dshm" (OuterVolumeSpecName: "dshm") pod "299f248b-9b51-4440-875a-c8e1ab8098a6" (UID: "299f248b-9b51-4440-875a-c8e1ab8098a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:29.639670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.639610    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299f248b-9b51-4440-875a-c8e1ab8098a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "299f248b-9b51-4440-875a-c8e1ab8098a6" (UID: "299f248b-9b51-4440-875a-c8e1ab8098a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:49:29.639670 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.639659    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299f248b-9b51-4440-875a-c8e1ab8098a6-kube-api-access-vgr6q" (OuterVolumeSpecName: "kube-api-access-vgr6q") pod "299f248b-9b51-4440-875a-c8e1ab8098a6" (UID: "299f248b-9b51-4440-875a-c8e1ab8098a6"). InnerVolumeSpecName "kube-api-access-vgr6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:49:29.738844 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.738801    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/299f248b-9b51-4440-875a-c8e1ab8098a6-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:29.738844 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.738839    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgr6q\" (UniqueName: \"kubernetes.io/projected/299f248b-9b51-4440-875a-c8e1ab8098a6-kube-api-access-vgr6q\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:29.738844 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.738849    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:29.738844 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.738857    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/299f248b-9b51-4440-875a-c8e1ab8098a6-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:29.755890 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.755866    2567 generic.go:356] "Generic (PLEG): container finished" podID="299f248b-9b51-4440-875a-c8e1ab8098a6" containerID="74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19" exitCode=0
Jul 07 19:49:29.756045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.755910    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"
Jul 07 19:49:29.756045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.755954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9" event={"ID":"299f248b-9b51-4440-875a-c8e1ab8098a6","Type":"ContainerDied","Data":"74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19"}
Jul 07 19:49:29.756045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.755997    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9" event={"ID":"299f248b-9b51-4440-875a-c8e1ab8098a6","Type":"ContainerDied","Data":"df6bfcab3216a6c53401224886483fc5fc04caac1c504cbbd98579e29e366927"}
Jul 07 19:49:29.756045 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.756016    2567 scope.go:117] "RemoveContainer" containerID="74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19"
Jul 07 19:49:29.757618 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.757595    2567 generic.go:356] "Generic (PLEG): container finished" podID="e799d733-4ad2-422d-8298-1845d7b988c1" containerID="2adb5b683d53d830abdb5bf75ab1004585c9f637dd421effb03e05ddb399a030" exitCode=0
Jul 07 19:49:29.757714 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.757629    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517756512" event={"ID":"e799d733-4ad2-422d-8298-1845d7b988c1","Type":"ContainerDied","Data":"2adb5b683d53d830abdb5bf75ab1004585c9f637dd421effb03e05ddb399a030"}
Jul 07 19:49:29.757714 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.757656    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517756512" event={"ID":"e799d733-4ad2-422d-8298-1845d7b988c1","Type":"ContainerStarted","Data":"a5f6e6b433125d4c1b0ebc2a840b018f1e5e78056de23cf24e0211583ab3f55f"}
Jul 07 19:49:29.764558 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.764542    2567 scope.go:117] "RemoveContainer" containerID="74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19"
Jul 07 19:49:29.764817 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:49:29.764797    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19\": container with ID starting with 74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19 not found: ID does not exist" containerID="74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19"
Jul 07 19:49:29.764862 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.764826    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19"} err="failed to get container status \"74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19\": rpc error: code = NotFound desc = could not find container \"74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19\": container with ID starting with 74c851b5cb507438f2bbb15fed001215913cce73bbb52767aed2b1bbf69d0a19 not found: ID does not exist"
Jul 07 19:49:29.820316 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.820251    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"]
Jul 07 19:49:29.830758 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.830733    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-778336/rate-test-model-778336-kserve-69db65b7bc-5qcs9"]
Jul 07 19:49:29.849635 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:29.849615    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="299f248b-9b51-4440-875a-c8e1ab8098a6" path="/var/lib/kubelet/pods/299f248b-9b51-4440-875a-c8e1ab8098a6/volumes"
Jul 07 19:49:30.105574 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.105484    2567 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq" podUID="f725e886-287b-40c2-9627-3b0142575695" containerName="main" containerID="cri-o://c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72" gracePeriod=45
Jul 07 19:49:30.355070 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.355042    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:49:30.443873 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.443811    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6hlg\" (UniqueName: \"kubernetes.io/projected/f725e886-287b-40c2-9627-3b0142575695-kube-api-access-l6hlg\") pod \"f725e886-287b-40c2-9627-3b0142575695\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") "
Jul 07 19:49:30.443873 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.443866    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-tmp-dir\") pod \"f725e886-287b-40c2-9627-3b0142575695\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") "
Jul 07 19:49:30.444062 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.443888    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-model-cache\") pod \"f725e886-287b-40c2-9627-3b0142575695\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") "
Jul 07 19:49:30.444062 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.443939    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-dshm\") pod \"f725e886-287b-40c2-9627-3b0142575695\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") "
Jul 07 19:49:30.444062 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.443961    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-home\") pod \"f725e886-287b-40c2-9627-3b0142575695\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") "
Jul 07 19:49:30.444062 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.443991    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f725e886-287b-40c2-9627-3b0142575695-tls-certs\") pod \"f725e886-287b-40c2-9627-3b0142575695\" (UID: \"f725e886-287b-40c2-9627-3b0142575695\") "
Jul 07 19:49:30.444270 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.444181    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-model-cache" (OuterVolumeSpecName: "model-cache") pod "f725e886-287b-40c2-9627-3b0142575695" (UID: "f725e886-287b-40c2-9627-3b0142575695"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:30.444270 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.444193    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f725e886-287b-40c2-9627-3b0142575695" (UID: "f725e886-287b-40c2-9627-3b0142575695"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:30.444270 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.444212    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-home" (OuterVolumeSpecName: "home") pod "f725e886-287b-40c2-9627-3b0142575695" (UID: "f725e886-287b-40c2-9627-3b0142575695"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:30.446056 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.446027    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-dshm" (OuterVolumeSpecName: "dshm") pod "f725e886-287b-40c2-9627-3b0142575695" (UID: "f725e886-287b-40c2-9627-3b0142575695"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 19:49:30.446056 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.446037    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f725e886-287b-40c2-9627-3b0142575695-kube-api-access-l6hlg" (OuterVolumeSpecName: "kube-api-access-l6hlg") pod "f725e886-287b-40c2-9627-3b0142575695" (UID: "f725e886-287b-40c2-9627-3b0142575695"). InnerVolumeSpecName "kube-api-access-l6hlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:49:30.446200 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.446083    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f725e886-287b-40c2-9627-3b0142575695-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f725e886-287b-40c2-9627-3b0142575695" (UID: "f725e886-287b-40c2-9627-3b0142575695"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 19:49:30.545393 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.545363    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-tmp-dir\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:30.545393 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.545387    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-model-cache\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:30.545393 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.545400    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-dshm\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:30.545581 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.545411    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f725e886-287b-40c2-9627-3b0142575695-home\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:30.545581 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.545421    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f725e886-287b-40c2-9627-3b0142575695-tls-certs\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:30.545581 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.545433    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6hlg\" (UniqueName: \"kubernetes.io/projected/f725e886-287b-40c2-9627-3b0142575695-kube-api-access-l6hlg\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:30.764912 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.764875    2567 generic.go:356] "Generic (PLEG): container finished" podID="f725e886-287b-40c2-9627-3b0142575695" containerID="c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72" exitCode=0
Jul 07 19:49:30.765077 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.764945    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"
Jul 07 19:49:30.765077 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.764944    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq" event={"ID":"f725e886-287b-40c2-9627-3b0142575695","Type":"ContainerDied","Data":"c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72"}
Jul 07 19:49:30.765077 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.764979    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq" event={"ID":"f725e886-287b-40c2-9627-3b0142575695","Type":"ContainerDied","Data":"2bd6ca60e1e64f32772bc67b2d1fca8cfe7a4dc54da009146dc30f623cd4813c"}
Jul 07 19:49:30.765077 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.765002    2567 scope.go:117] "RemoveContainer" containerID="c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72"
Jul 07 19:49:30.775550 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.775523    2567 scope.go:117] "RemoveContainer" containerID="c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72"
Jul 07 19:49:30.777791 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:49:30.777716    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72\": container with ID starting with c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72 not found: ID does not exist" containerID="c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72"
Jul 07 19:49:30.777791 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.777761    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72"} err="failed to get container status \"c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72\": rpc error: code = NotFound desc = could not find container \"c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72\": container with ID starting with c46609ded5b00b22d99cc6ede88e970515915c861cd3c5a94c9b8e73b1dfcd72 not found: ID does not exist"
Jul 07 19:49:30.793635 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.793606    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"]
Jul 07 19:49:30.796321 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.796293    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fe4a68/rate-test-model-fe4a68-kserve-78ff6568bc-z6hhq"]
Jul 07 19:49:30.913311 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.913288    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:30.938426 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.938397    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7392-140448517756512_e799d733-4ad2-422d-8298-1845d7b988c1/test-curl-7392-140448517756512/0.log"
Jul 07 19:49:30.965877 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.965851    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:30.969070 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:30.969050    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:31.049881 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.049823    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clw6x\" (UniqueName: \"kubernetes.io/projected/e799d733-4ad2-422d-8298-1845d7b988c1-kube-api-access-clw6x\") pod \"e799d733-4ad2-422d-8298-1845d7b988c1\" (UID: \"e799d733-4ad2-422d-8298-1845d7b988c1\") "
Jul 07 19:49:31.051842 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.051816    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e799d733-4ad2-422d-8298-1845d7b988c1-kube-api-access-clw6x" (OuterVolumeSpecName: "kube-api-access-clw6x") pod "e799d733-4ad2-422d-8298-1845d7b988c1" (UID: "e799d733-4ad2-422d-8298-1845d7b988c1"). InnerVolumeSpecName "kube-api-access-clw6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:49:31.099371 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099342    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:31.099620 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099609    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e799d733-4ad2-422d-8298-1845d7b988c1" containerName="test-curl-7392-140448517756512"
Jul 07 19:49:31.099658 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099622    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e799d733-4ad2-422d-8298-1845d7b988c1" containerName="test-curl-7392-140448517756512"
Jul 07 19:49:31.099658 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099635    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299f248b-9b51-4440-875a-c8e1ab8098a6" containerName="main"
Jul 07 19:49:31.099658 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099640    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="299f248b-9b51-4440-875a-c8e1ab8098a6" containerName="main"
Jul 07 19:49:31.099658 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099655    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f725e886-287b-40c2-9627-3b0142575695" containerName="main"
Jul 07 19:49:31.099780 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099660    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f725e886-287b-40c2-9627-3b0142575695" containerName="main"
Jul 07 19:49:31.099780 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099708    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f725e886-287b-40c2-9627-3b0142575695" containerName="main"
Jul 07 19:49:31.099780 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099713    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e799d733-4ad2-422d-8298-1845d7b988c1" containerName="test-curl-7392-140448517756512"
Jul 07 19:49:31.099780 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.099721    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="299f248b-9b51-4440-875a-c8e1ab8098a6" containerName="main"
Jul 07 19:49:31.102529 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.102514    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:31.113939 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.113904    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:31.151298 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.151275    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clw6x\" (UniqueName: \"kubernetes.io/projected/e799d733-4ad2-422d-8298-1845d7b988c1-kube-api-access-clw6x\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:31.252137 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.252107    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq462\" (UniqueName: \"kubernetes.io/projected/0c05381f-a26b-42d2-a978-40ce2d9d5ea1-kube-api-access-cq462\") pod \"test-curl-7392-140448517756512\" (UID: \"0c05381f-a26b-42d2-a978-40ce2d9d5ea1\") " pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:31.352974 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.352896    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq462\" (UniqueName: \"kubernetes.io/projected/0c05381f-a26b-42d2-a978-40ce2d9d5ea1-kube-api-access-cq462\") pod \"test-curl-7392-140448517756512\" (UID: \"0c05381f-a26b-42d2-a978-40ce2d9d5ea1\") " pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:31.362829 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.362807    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq462\" (UniqueName: \"kubernetes.io/projected/0c05381f-a26b-42d2-a978-40ce2d9d5ea1-kube-api-access-cq462\") pod \"test-curl-7392-140448517756512\" (UID: \"0c05381f-a26b-42d2-a978-40ce2d9d5ea1\") " pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:31.411338 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.411303    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:31.545222 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.545197    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:31.546810 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:49:31.546785    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c05381f_a26b_42d2_a978_40ce2d9d5ea1.slice/crio-59efce671c9181026667142a79693f6d30a0b1ae9a44eb0b9c0dc2ec42def96a WatchSource:0}: Error finding container 59efce671c9181026667142a79693f6d30a0b1ae9a44eb0b9c0dc2ec42def96a: Status 404 returned error can't find the container with id 59efce671c9181026667142a79693f6d30a0b1ae9a44eb0b9c0dc2ec42def96a
Jul 07 19:49:31.771326 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.771297    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517756512" event={"ID":"0c05381f-a26b-42d2-a978-40ce2d9d5ea1","Type":"ContainerStarted","Data":"59efce671c9181026667142a79693f6d30a0b1ae9a44eb0b9c0dc2ec42def96a"}
Jul 07 19:49:31.772395 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.772378    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5f6e6b433125d4c1b0ebc2a840b018f1e5e78056de23cf24e0211583ab3f55f"
Jul 07 19:49:31.772487 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.772407    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:31.783896 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.783858    2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7392-140448517756512" oldPodUID="e799d733-4ad2-422d-8298-1845d7b988c1" podUID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1"
Jul 07 19:49:31.850131 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.850108    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e799d733-4ad2-422d-8298-1845d7b988c1" path="/var/lib/kubelet/pods/e799d733-4ad2-422d-8298-1845d7b988c1/volumes"
Jul 07 19:49:31.850483 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:31.850470    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f725e886-287b-40c2-9627-3b0142575695" path="/var/lib/kubelet/pods/f725e886-287b-40c2-9627-3b0142575695/volumes"
Jul 07 19:49:32.778163 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:32.778122    2567 generic.go:356] "Generic (PLEG): container finished" podID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1" containerID="347d88870bf53b764d91ef6fa57a4f3ee36b7b2faebd368c5113dff6b619e51e" exitCode=0
Jul 07 19:49:32.778523 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:32.778183    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7392-140448517756512" event={"ID":"0c05381f-a26b-42d2-a978-40ce2d9d5ea1","Type":"ContainerDied","Data":"347d88870bf53b764d91ef6fa57a4f3ee36b7b2faebd368c5113dff6b619e51e"}
Jul 07 19:49:33.915535 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:33.915505    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:33.948561 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:33.948522    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7392-140448517756512_0c05381f-a26b-42d2-a978-40ce2d9d5ea1/test-curl-7392-140448517756512/0.log"
Jul 07 19:49:33.985972 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:33.985947    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:33.996875 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:33.996850    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7392-140448517756512"]
Jul 07 19:49:34.076871 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:34.076800    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq462\" (UniqueName: \"kubernetes.io/projected/0c05381f-a26b-42d2-a978-40ce2d9d5ea1-kube-api-access-cq462\") pod \"0c05381f-a26b-42d2-a978-40ce2d9d5ea1\" (UID: \"0c05381f-a26b-42d2-a978-40ce2d9d5ea1\") "
Jul 07 19:49:34.078911 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:34.078892    2567 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c05381f-a26b-42d2-a978-40ce2d9d5ea1-kube-api-access-cq462" (OuterVolumeSpecName: "kube-api-access-cq462") pod "0c05381f-a26b-42d2-a978-40ce2d9d5ea1" (UID: "0c05381f-a26b-42d2-a978-40ce2d9d5ea1"). InnerVolumeSpecName "kube-api-access-cq462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 19:49:34.178017 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:34.177991    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq462\" (UniqueName: \"kubernetes.io/projected/0c05381f-a26b-42d2-a978-40ce2d9d5ea1-kube-api-access-cq462\") on node \"ip-10-0-141-187.ec2.internal\" DevicePath \"\""
Jul 07 19:49:34.787292 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:34.787266    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59efce671c9181026667142a79693f6d30a0b1ae9a44eb0b9c0dc2ec42def96a"
Jul 07 19:49:34.787449 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:34.787300    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:34.799439 ip-10-0-141-187 kubenswrapper[2567]: E0707 19:49:34.799413    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7392-140448517756512\" is forbidden: User \"system:node:ip-10-0-141-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-187.ec2.internal' and this object" podUID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1" pod="opendatahub/test-curl-7392-140448517756512"
Jul 07 19:49:35.851047 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:35.851017    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1" path="/var/lib/kubelet/pods/0c05381f-a26b-42d2-a978-40ce2d9d5ea1/volumes"
Jul 07 19:49:51.767004 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:51.766909    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7444d9f889-z8wqf_eec2c6d1-42d7-4e07-b7bb-82875e16a5ae/authorino/0.log"
Jul 07 19:49:57.155058 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:57.155025    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wxxct_7f298848-7828-4177-8d19-09b8a4954e27/server/0.log"
Jul 07 19:49:57.277563 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:57.277536    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sz472_e88debc4-d94c-4551-a194-d792ee4d5117/manager/0.log"
Jul 07 19:49:57.524698 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:57.524664    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cwbjb_a961a3a6-ed86-43cf-8678-4d01a77ba4aa/manager/0.log"
Jul 07 19:49:57.531422 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:57.531403    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cwbjb_a961a3a6-ed86-43cf-8678-4d01a77ba4aa/copy-manifests/0.log"
Jul 07 19:49:59.231784 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:59.231751    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7444d9f889-z8wqf_eec2c6d1-42d7-4e07-b7bb-82875e16a5ae/authorino/0.log"
Jul 07 19:49:59.628976 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:49:59.628901    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kd7sm_b7586f04-2b32-4bc1-816f-fcd58e784f98/kuadrant-console-plugin/0.log"
Jul 07 19:50:00.933237 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:00.933209    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-777c59bd5f-pws4w_9cd66246-00db-4bfa-ad22-f598dff937d4/kube-auth-proxy/0.log"
Jul 07 19:50:01.354437 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:01.354395    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sz4zv_e740858b-e6ec-47b8-892f-c70d7c647cea/payload-processing/0.log"
Jul 07 19:50:02.350793 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:02.350704    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7nfvn_67f740a4-4b87-4f2a-960f-3b810fb5ff66/main/0.log"
Jul 07 19:50:02.470540 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:02.470506    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-czj5q_d1b417a3-7e78-41ef-bc7a-281d02b4b467/main/0.log"
Jul 07 19:50:07.058970 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.058933    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jpkhx/must-gather-8cbhw"]
Jul 07 19:50:07.059326 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.059231    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1" containerName="test-curl-7392-140448517756512"
Jul 07 19:50:07.059326 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.059242    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1" containerName="test-curl-7392-140448517756512"
Jul 07 19:50:07.059326 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.059309    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c05381f-a26b-42d2-a978-40ce2d9d5ea1" containerName="test-curl-7392-140448517756512"
Jul 07 19:50:07.062395 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.062374    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.065306 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.065283    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpkhx\"/\"openshift-service-ca.crt\""
Jul 07 19:50:07.065398 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.065381    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jpkhx\"/\"kube-root-ca.crt\""
Jul 07 19:50:07.066156 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.066139    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jpkhx\"/\"default-dockercfg-x9f5v\""
Jul 07 19:50:07.073686 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.073662    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpkhx/must-gather-8cbhw"]
Jul 07 19:50:07.127131 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.127100    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lr9m\" (UniqueName: \"kubernetes.io/projected/95c6b23d-c5cf-40a5-a4a4-537253d403fc-kube-api-access-6lr9m\") pod \"must-gather-8cbhw\" (UID: \"95c6b23d-c5cf-40a5-a4a4-537253d403fc\") " pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.127250 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.127141    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95c6b23d-c5cf-40a5-a4a4-537253d403fc-must-gather-output\") pod \"must-gather-8cbhw\" (UID: \"95c6b23d-c5cf-40a5-a4a4-537253d403fc\") " pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.228193 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.228163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lr9m\" (UniqueName: \"kubernetes.io/projected/95c6b23d-c5cf-40a5-a4a4-537253d403fc-kube-api-access-6lr9m\") pod \"must-gather-8cbhw\" (UID: \"95c6b23d-c5cf-40a5-a4a4-537253d403fc\") " pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.228314 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.228204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95c6b23d-c5cf-40a5-a4a4-537253d403fc-must-gather-output\") pod \"must-gather-8cbhw\" (UID: \"95c6b23d-c5cf-40a5-a4a4-537253d403fc\") " pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.228505 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.228484    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/95c6b23d-c5cf-40a5-a4a4-537253d403fc-must-gather-output\") pod \"must-gather-8cbhw\" (UID: \"95c6b23d-c5cf-40a5-a4a4-537253d403fc\") " pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.247489 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.247458    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lr9m\" (UniqueName: \"kubernetes.io/projected/95c6b23d-c5cf-40a5-a4a4-537253d403fc-kube-api-access-6lr9m\") pod \"must-gather-8cbhw\" (UID: \"95c6b23d-c5cf-40a5-a4a4-537253d403fc\") " pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.371020 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.370937    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpkhx/must-gather-8cbhw"
Jul 07 19:50:07.503140 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.503083    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpkhx/must-gather-8cbhw"]
Jul 07 19:50:07.505906 ip-10-0-141-187 kubenswrapper[2567]: W0707 19:50:07.505880    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95c6b23d_c5cf_40a5_a4a4_537253d403fc.slice/crio-ea00f14a1136820ebd8a1a3397eb6c8f32e6cd146ff517464c33af333227a211 WatchSource:0}: Error finding container ea00f14a1136820ebd8a1a3397eb6c8f32e6cd146ff517464c33af333227a211: Status 404 returned error can't find the container with id ea00f14a1136820ebd8a1a3397eb6c8f32e6cd146ff517464c33af333227a211
Jul 07 19:50:07.925386 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:07.925349    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/must-gather-8cbhw" event={"ID":"95c6b23d-c5cf-40a5-a4a4-537253d403fc","Type":"ContainerStarted","Data":"ea00f14a1136820ebd8a1a3397eb6c8f32e6cd146ff517464c33af333227a211"}
Jul 07 19:50:08.933009 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:08.932972    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/must-gather-8cbhw" event={"ID":"95c6b23d-c5cf-40a5-a4a4-537253d403fc","Type":"ContainerStarted","Data":"8cf1bfae3e3cba393ae3257a54696715dc152ab58a4d9b1d3f40251e5d9359e4"}
Jul 07 19:50:08.933499 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:08.933018    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/must-gather-8cbhw" event={"ID":"95c6b23d-c5cf-40a5-a4a4-537253d403fc","Type":"ContainerStarted","Data":"e7f83db6adac4c25a534b02e31f7bb52cf7f123150db9c5a692982d27022593d"}
Jul 07 19:50:08.953635 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:08.953571    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jpkhx/must-gather-8cbhw" podStartSLOduration=0.994351737 podStartE2EDuration="1.953551174s" totalImagesPullingTime="959.199437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:50:08.95223291 +0000 UTC m=+2529.603858845" watchObservedRunningTime="2026-07-07 19:50:08.953551174 +0000 UTC m=+2529.605177110"
Jul 07 19:50:10.355234 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:10.355195    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cwklg_d9b59752-23f5-462c-a926-8716ec8155d2/global-pull-secret-syncer/0.log"
Jul 07 19:50:10.584102 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:10.584065    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zkc97_0b3914eb-b48e-4af2-a65e-0d759c315c29/konnectivity-agent/0.log"
Jul 07 19:50:10.731963 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:10.731936    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-187.ec2.internal_b469ca5e754e40f9721a7a95476e077f/haproxy/0.log"
Jul 07 19:50:15.925403 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:15.923731    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7444d9f889-z8wqf_eec2c6d1-42d7-4e07-b7bb-82875e16a5ae/authorino/0.log"
Jul 07 19:50:16.044537 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:16.044503    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kd7sm_b7586f04-2b32-4bc1-816f-fcd58e784f98/kuadrant-console-plugin/0.log"
Jul 07 19:50:18.719430 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:18.719395    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hc7l5_1f0fb3cb-16c8-4007-a11d-6ec52eb5370f/node-exporter/0.log"
Jul 07 19:50:18.758457 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:18.758393    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hc7l5_1f0fb3cb-16c8-4007-a11d-6ec52eb5370f/kube-rbac-proxy/0.log"
Jul 07 19:50:18.787851 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:18.787825    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hc7l5_1f0fb3cb-16c8-4007-a11d-6ec52eb5370f/init-textfile/0.log"
Jul 07 19:50:19.977390 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:19.977347    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"]
Jul 07 19:50:19.985096 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:19.985069    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.016389 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.016357    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"]
Jul 07 19:50:20.062284 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.062248    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-sys\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.062284 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.062296    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-lib-modules\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.062542 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.062323    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-podres\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.062542 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.062437    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5rj\" (UniqueName: \"kubernetes.io/projected/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-kube-api-access-vq5rj\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.062542 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.062467    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-proc\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.163772 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163715    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5rj\" (UniqueName: \"kubernetes.io/projected/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-kube-api-access-vq5rj\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.163772 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163772    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-proc\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163823    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-sys\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163853    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-lib-modules\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-podres\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163978    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-sys\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164030 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.163981    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-proc\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164289 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.164054    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-podres\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.164289 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.164062    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-lib-modules\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.173388 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.173355    2567 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5rj\" (UniqueName: \"kubernetes.io/projected/71917c7e-dc75-4f8e-abbb-2e8b437fa1ed-kube-api-access-vq5rj\") pod \"perf-node-gather-daemonset-xcvdc\" (UID: \"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed\") " pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.296076 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.295991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:20.483233 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:20.481519    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"]
Jul 07 19:50:21.010035 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:21.009991    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc" event={"ID":"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed","Type":"ContainerStarted","Data":"729c8285939c28350d099bcc6db4cf9481beb37e5fec5a0d821618f85a142997"}
Jul 07 19:50:21.010673 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:21.010096    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc" event={"ID":"71917c7e-dc75-4f8e-abbb-2e8b437fa1ed","Type":"ContainerStarted","Data":"7175625b0c9303ccf8189d15debd1b9a3f48ea82d37c6b4d2a95b0072d50ea00"}
Jul 07 19:50:21.010673 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:21.010148    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:21.050456 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:21.050404    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc" podStartSLOduration=2.050387314 podStartE2EDuration="2.050387314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:50:21.050103745 +0000 UTC m=+2541.701729678" watchObservedRunningTime="2026-07-07 19:50:21.050387314 +0000 UTC m=+2541.702013246"
Jul 07 19:50:24.028801 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:24.028771    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nsjg_3a46f326-2959-42bf-814d-bc4364f9a41d/dns/0.log"
Jul 07 19:50:24.068159 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:24.068130    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7nsjg_3a46f326-2959-42bf-814d-bc4364f9a41d/kube-rbac-proxy/0.log"
Jul 07 19:50:24.274829 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:24.274804    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6kfr4_4acae846-0bb0-41fa-af29-3ec6eeb83eee/dns-node-resolver/0.log"
Jul 07 19:50:24.984397 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:24.984357    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b87dd9f8b-zxwpl_09ba6eff-2549-4380-9f07-9e2ee46720b0/registry/0.log"
Jul 07 19:50:25.029006 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:25.028905    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5hw59_8700ea39-999a-468b-b53e-5cabbcc4b5ed/node-ca/0.log"
Jul 07 19:50:26.501095 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:26.501067    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-777c59bd5f-pws4w_9cd66246-00db-4bfa-ad22-f598dff937d4/kube-auth-proxy/0.log"
Jul 07 19:50:26.761818 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:26.761735    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sz4zv_e740858b-e6ec-47b8-892f-c70d7c647cea/payload-processing/0.log"
Jul 07 19:50:27.026327 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:27.026253    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jpkhx/perf-node-gather-daemonset-xcvdc"
Jul 07 19:50:27.531951 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:27.531906    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jg7vz_a9b16b18-ef1e-49c5-ad84-f1e01935e6c6/serve-healthcheck-canary/0.log"
Jul 07 19:50:28.430709 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:28.430657    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwrdf_3cbbecc2-3a6b-4e03-9060-efc6f1675ae2/kube-rbac-proxy/0.log"
Jul 07 19:50:28.468507 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:28.468477    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwrdf_3cbbecc2-3a6b-4e03-9060-efc6f1675ae2/exporter/0.log"
Jul 07 19:50:28.518949 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:28.518903    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nwrdf_3cbbecc2-3a6b-4e03-9060-efc6f1675ae2/extractor/0.log"
Jul 07 19:50:30.544679 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:30.544652    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-777c59bd5f-pws4w_9cd66246-00db-4bfa-ad22-f598dff937d4/kube-auth-proxy/0.log"
Jul 07 19:50:30.734514 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:30.734477    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sz4zv_e740858b-e6ec-47b8-892f-c70d7c647cea/payload-processing/0.log"
Jul 07 19:50:31.732717 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:31.732688    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-wxxct_7f298848-7828-4177-8d19-09b8a4954e27/server/0.log"
Jul 07 19:50:31.862136 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:31.862105    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sz472_e88debc4-d94c-4551-a194-d792ee4d5117/manager/0.log"
Jul 07 19:50:31.979864 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:31.979831    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cwbjb_a961a3a6-ed86-43cf-8678-4d01a77ba4aa/manager/0.log"
Jul 07 19:50:32.028576 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:32.028502    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8689849645-cwbjb_a961a3a6-ed86-43cf-8678-4d01a77ba4aa/copy-manifests/0.log"
Jul 07 19:50:32.331495 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:32.331371    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-777c59bd5f-pws4w_9cd66246-00db-4bfa-ad22-f598dff937d4/kube-auth-proxy/0.log"
Jul 07 19:50:32.532223 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:32.532188    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-sz4zv_e740858b-e6ec-47b8-892f-c70d7c647cea/payload-processing/0.log"
Jul 07 19:50:33.646276 ip-10-0-141-187 kubenswrapper[2567]: I0707 19:50:33.646237    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d95ccd586-j7c9m_2c1fea3b-36df-4217-865b-8b8c1bdfed23/manager/0.log"