Jul 14 00:57:45.364612 ip-10-0-138-193 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 00:57:45.364624 ip-10-0-138-193 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 00:57:45.364635 ip-10-0-138-193 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 00:57:45.364944 ip-10-0-138-193 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 00:57:55.401316 ip-10-0-138-193 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 00:57:55.401336 ip-10-0-138-193 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 22c2fb71ba324d1b82b9d371287957ed --
Jul 14 00:59:56.929751 ip-10-0-138-193 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 00:59:57.345941 ip-10-0-138-193 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:59:57.345941 ip-10-0-138-193 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 00:59:57.345941 ip-10-0-138-193 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:59:57.345941 ip-10-0-138-193 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 00:59:57.345941 ip-10-0-138-193 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:59:57.346757 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.346613    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 00:59:57.350255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350240    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350256    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350262    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350264    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350267    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350270    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350273    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350275    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350278    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350281    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350284    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350286    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350289    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350292    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:57.350291 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350296    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350300    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350303    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350306    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350308    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350311    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350313    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350317    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350320    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350322    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350325    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350327    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350330    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350332    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350335    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350337    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350340    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350342    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350345    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350347    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:57.350651 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350350    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350352    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350355    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350357    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350359    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350362    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350364    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350367    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350369    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350380    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350383    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350385    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350388    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350390    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350393    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350396    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350399    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350402    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350404    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:57.351131 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350407    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350410    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350412    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350415    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350417    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350419    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350423    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350439    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350441    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350444    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350446    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350449    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350452    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350454    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350457    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350460    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350462    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350465    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350469    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350471    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:57.351603 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350473    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350476    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350478    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350482    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350487    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350490    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350494    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350497    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350500    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350503    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350506    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350508    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350511    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350514    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350517    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350519    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350522    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350525    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350528    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350530    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:57.352082 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350533    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350535    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350538    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350541    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350544    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.350547    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351239    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351246    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351250    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351252    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351255    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351258    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351261    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351263    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351266    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351270    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351273    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351276    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351280    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:57.352568 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351283    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351285    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351288    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351290    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351293    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351295    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351298    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351300    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351302    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351305    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351308    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351310    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351313    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351316    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351318    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351321    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351325    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351327    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351330    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351332    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:57.353028 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351335    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351337    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351339    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351342    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351344    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351347    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351349    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351352    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351354    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351357    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351359    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351362    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351365    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351367    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351370    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351372    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351375    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351377    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351379    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:57.353525 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351382    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351384    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351388    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351392    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351395    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351397    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351400    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351404    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351407    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351410    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351412    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351415    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351417    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351419    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351422    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351442    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351445    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351447    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351450    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:57.353985 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351453    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351455    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351458    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351460    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351463    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351466    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351468    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351471    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351473    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351476    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351480    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351482    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351484    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351487    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351489    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351492    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351494    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351497    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351499    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351502    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:57.354467 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351504    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351506    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351509    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351511    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351514    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351516    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351519    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351522    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351871    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351876    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351879    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351882    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351885    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351888    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351891    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351893    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351896    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351899    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351902    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351904    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:57.354968 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351907    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351909    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351912    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351915    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351918    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351920    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351923    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351925    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351927    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351930    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351932    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351935    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351937    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351940    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351942    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351944    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351947    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351949    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351952    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351954    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:57.355471 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351957    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351961    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351964    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351966    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351968    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351971    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351974    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351976    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351978    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351981    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351984    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351986    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351989    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351991    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351994    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351997    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.351999    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352001    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352004    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352006    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:57.355975 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352009    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352011    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352013    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352016    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352018    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352020    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352025    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352028    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352031    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352034    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352037    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352039    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352042    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352044    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352047    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352050    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352052    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352058    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352061    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352064    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:57.356580 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352066    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352069    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352072    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352074    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352077    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352080    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352083    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352087    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352090    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352093    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352096    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352099    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352101    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352104    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352107    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352110    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352113    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352115    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352117    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:57.357061 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352120    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352123    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352125    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352128    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352130    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352132    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352135    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352137    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352214    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352226    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352236    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352243    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352250    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352254    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352261    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352265    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352269    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352272    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352275    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352278    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352281    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352284    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 14 00:59:57.357554 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352286    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352289    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352292    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352295    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352299    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352301    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352304    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352307    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352310    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352314    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352316    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352319    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352322    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352325    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352329    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352332    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352335    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352338    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352341    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352346    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352351    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352353    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352356    2578 flags.go:64] FLAG: --enable-server="true"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352360    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352364    2578 flags.go:64] FLAG: --event-burst="100"
Jul 14 00:59:57.358079 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352367    2578 flags.go:64] FLAG: --event-qps="50"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352370    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352373    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352375    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352379    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352381    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352384    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352387    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352390    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352393    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352400    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352403    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352406    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352408    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352411    2578 flags.go:64] FLAG: --feature-gates=""
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352415    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352418    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352421    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352437    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352441    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352444    2578 flags.go:64] FLAG: --help="false"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352447    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352450    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352453    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 00:59:57.358703 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352455    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352459    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352467    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352471    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352474    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352476    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352479    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352482    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352485    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352488    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352491    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352494    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352497    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352499    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352502    2578 flags.go:64] FLAG: --lock-file=""
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352505    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352507    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352510    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352515    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352518    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352520    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352523    2578 flags.go:64] FLAG: --logging-format="text"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352526    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352529    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 00:59:57.359299 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352532    2578 flags.go:64] FLAG: --manifest-url=""
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352535    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352539    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352541    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352550    2578 flags.go:64] FLAG: --max-pods="110"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352552    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352555    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352558    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352561    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352563    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352568    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352572    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352579    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352582    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352585    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352589    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352592    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352598    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352600    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352604    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352607    2578 flags.go:64] FLAG: --port="10250"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352609    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352612    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0181d736e2ebab029"
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352615    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 14 00:59:57.359978 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352618    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352621    2578 flags.go:64] FLAG: --register-node="true"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352624    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352627    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352630    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352632    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352635    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352638    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352641    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352644    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352647    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352649    2578 flags.go:64] FLAG: --runonce="false"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352652    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352655    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352658    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352660    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352663    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352666    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352670    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352673    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352677    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352679    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352682    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352685    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352687    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 00:59:57.360577 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352691    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352694    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352699    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352701    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352704    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352708    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352710    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352713    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352716    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352718    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352721    2578 flags.go:64] FLAG: --v="2"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352725    2578 flags.go:64] FLAG: --version="false"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352729    2578 flags.go:64] FLAG: --vmodule=""
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352732    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.352735    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352848    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352851    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352854    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352857    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352861    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352864    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352867    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:57.361204 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352870    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352874    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352877    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352881    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352884    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352887    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352890    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352894    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352897    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352899    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352902    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352906    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352908    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352911    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352913    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352916    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352918    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352921    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352923    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:57.361733 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352926    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352928    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352930    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352933    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352935    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352938    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352940    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352943    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352945    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352948    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352950    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352953    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352955    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352958    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352960    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352963    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352966    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352969    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352971    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352974    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:57.362190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352976    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352978    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352981    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352983    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352986    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352989    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352991    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352993    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352995    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.352998    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353000    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353003    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353005    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353007    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353010    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353012    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353015    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353017    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353019    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353022    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:57.362713 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353024    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353027    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353029    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353032    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353035    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353037    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353040    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353042    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353046    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353048    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353051    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353053    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353056    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353058    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353061    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353063    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353066    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353068    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353071    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353073    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:57.363335 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353076    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353078    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353080    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353083    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353085    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353088    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353090    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353092    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353095    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353097    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353099    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353101    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.353104    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.354231    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361080    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 00:59:57.364146 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361095    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361115    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361121    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361176    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361182    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361185    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361188    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361191    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361195    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361199    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361203    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361207    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361210    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361213    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361216    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361219    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361221    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361224    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361227    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:57.364771 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361229    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361231    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361234    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361236    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361239    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361241    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361244    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361246    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361248    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361251    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361253    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361255    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361258    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361262    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361265    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361267    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361269    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361272    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361274    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361277    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:57.365228 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361279    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361281    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361284    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361286    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361289    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361291    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361293    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361295    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361298    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361300    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361303    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361305    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361307    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361310    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361312    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361315    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361317    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361319    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361322    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:57.365712 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361324    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361326    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361328    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361331    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361333    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361335    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361338    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361341    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361343    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361346    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361349    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361351    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361353    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361357    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361360    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361363    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361366    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361368    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361370    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:57.366200 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361373    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361375    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361377    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361380    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361382    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361384    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361388    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361391    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361394    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361396    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361399    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361402    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361404    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361407    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361409    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361411    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361414    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361416    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361418    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:57.366759 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361421    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361424    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361446    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361449    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361451    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361454    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361459    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361571    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361575    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361578    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361580    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361584    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361588    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361590    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361593    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:59:57.367207 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361595    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361598    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361600    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361603    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361607    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361610    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361612    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361615    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361617    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361620    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361622    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361624    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361627    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361630    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361634    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361636    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361639    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361642    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361645    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:59:57.367647 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361648    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361651    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361654    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361656    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361659    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361661    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361663    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361666    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361668    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361670    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361673    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361675    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361678    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361680    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361682    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361685    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361687    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361689    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361691    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361694    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:59:57.368190 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361696    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361698    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361701    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361703    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361705    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361708    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361710    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361712    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361715    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361718    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361720    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361723    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361725    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361728    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361731    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361733    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361736    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361738    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361741    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361743    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:59:57.368687 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361746    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361748    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361751    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361753    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361755    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361758    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361760    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361762    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361764    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361767    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361769    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361771    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361774    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361776    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361779    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361781    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361783    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361786    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361788    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361790    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:59:57.369255 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361792    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361795    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361798    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361800    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361803    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361805    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361807    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361810    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361812    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361815    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361817    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:57.361820    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361824    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.361928    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.365889    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 00:59:57.369943 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.367576    2578 server.go:1019] "Starting client certificate rotation"
Jul 14 00:59:57.370582 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.367698    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 00:59:57.370582 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.367749    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 00:59:57.370582 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.370127    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 00:59:57.371277 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.371255    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 00:59:57.380889 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.380872    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 14 00:59:57.382965 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.382943    2578 log.go:25] "Validated CRI v1 image API"
Jul 14 00:59:57.385955 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.385941    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 00:59:57.388594 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.388568    2578 fs.go:135] Filesystem UUIDs: map[53c456af-aa78-4acf-b632-a9a2b63c947d:/dev/nvme0n1p3 750e80e1-572d-4385-9bc5-1036d35c78fe:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 14 00:59:57.388680 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.388594    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 00:59:57.392731 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.392621    2578 manager.go:216] Machine: {Timestamp:2026-07-14 00:59:57.392139815 +0000 UTC m=+0.341661436 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099865 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a097ccbb1cfbb14b240b0d61cfd99 SystemUUID:ec2a097c-cbb1-cfbb-14b2-40b0d61cfd99 BootID:22c2fb71-ba32-4d1b-82b9-d371287957ed Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7c:b6:5f:6c:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7c:b6:5f:6c:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:d1:cb:80:35:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 00:59:57.392731 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.392721    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 00:59:57.392877 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.392820    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 00:59:57.394356 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.394300    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 00:59:57.394735 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.394713    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 00:59:57.394892 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.394489    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-193.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 00:59:57.395654 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.395641    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 00:59:57.395716 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.395659    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 00:59:57.395716 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.395669    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 00:59:57.395716 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.395695    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 00:59:57.397339 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.397326    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 00:59:57.401069 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401054    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 00:59:57.401195 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401184    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 00:59:57.401250 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401222    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 00:59:57.401250 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401237    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 00:59:57.401343 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401255    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 00:59:57.401343 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401268    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 14 00:59:57.401343 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401280    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 00:59:57.401657 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401643    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 00:59:57.401718 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.401666    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 00:59:57.404165 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.404150    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 00:59:57.406137 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.406123    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 00:59:57.406196 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.406154    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 00:59:57.407084 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407070    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407091    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407098    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407103    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407112    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407117    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407125    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407130    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407136    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 00:59:57.407144 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407141    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 00:59:57.407369 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407149    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 00:59:57.407369 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407154    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 00:59:57.407718 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407706    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 00:59:57.407772 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.407720    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 00:59:57.410664 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.410636    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-193.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 00:59:57.411259 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.411246    2578 server.go:1268] "Started kubelet"
Jul 14 00:59:57.411259 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.411249    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 00:59:57.411356 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.411249    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 00:59:57.411356 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.411325    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 00:59:57.411477 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.411370    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 00:59:57.411477 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.411448    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 00:59:57.412537 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.412517    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 00:59:57.412795 ip-10-0-138-193 systemd[1]: Started Kubernetes Kubelet.
Jul 14 00:59:57.414119 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.414106    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 00:59:57.416232 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.416216    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nnssf"
Jul 14 00:59:57.416471 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.416445    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 00:59:57.418027 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418009    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 00:59:57.418094 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418014    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 00:59:57.418217 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.418198    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.418790 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418776    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 00:59:57.418790 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418790    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 00:59:57.418925 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418777    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 00:59:57.418925 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418913    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 00:59:57.418925 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.418922    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 00:59:57.419472 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.419454    2578 factory.go:153] Registering CRI-O factory
Jul 14 00:59:57.419546 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.419474    2578 factory.go:223] Registration of the crio container factory successfully
Jul 14 00:59:57.422139 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.419537    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 00:59:57.422227 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.422144    2578 factory.go:55] Registering systemd factory
Jul 14 00:59:57.422227 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.422157    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 14 00:59:57.422227 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.422180    2578 factory.go:103] Registering Raw factory
Jul 14 00:59:57.422227 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.422196    2578 manager.go:1196] Started watching for new ooms in manager
Jul 14 00:59:57.422380 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.419159    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-193.ec2.internal.18c2022da242f322  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-193.ec2.internal,UID:ip-10-0-138-193.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-193.ec2.internal,},FirstTimestamp:2026-07-14 00:59:57.41121821 +0000 UTC m=+0.360739832,LastTimestamp:2026-07-14 00:59:57.41121821 +0000 UTC m=+0.360739832,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-193.ec2.internal,}"
Jul 14 00:59:57.422850 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.422834    2578 manager.go:349] Starting recovery of all containers
Jul 14 00:59:57.423672 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.423651    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nnssf"
Jul 14 00:59:57.429493 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.427408    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 00:59:57.429493 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.427580    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-193.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 00:59:57.434528 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.434396    2578 manager.go:354] Recovery completed
Jul 14 00:59:57.438685 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.438672    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.440358 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.440343    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.440406 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.440371    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.440406 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.440382    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.440949 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.440934    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 00:59:57.440949 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.440947    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.441084 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.440948    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 00:59:57.441084 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.441026    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 00:59:57.441817 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.441802    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.441867 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.441843    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.441867 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.441859    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.442206 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.442193    2578 policy_none.go:49] "None policy: Start"
Jul 14 00:59:57.442258 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.442213    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 00:59:57.442258 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.442227    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 00:59:57.443274 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.443261    2578 policy_none.go:47] "Start"
Jul 14 00:59:57.477935 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.477914    2578 manager.go:337] "Starting Device Plugin manager"
Jul 14 00:59:57.478029 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.477963    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 00:59:57.478029 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.477976    2578 server.go:87] "Starting device plugin registration server"
Jul 14 00:59:57.478210 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.478197    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 00:59:57.478263 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.478212    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 00:59:57.478327 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.478301    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 00:59:57.478417 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.478399    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 00:59:57.478417 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.478417    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 00:59:57.479004 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.478987    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 00:59:57.479099 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.479025    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.578630 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.578610    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.579611 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.579445    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.579611 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.579473    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.579611 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.579487    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.579611 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.579514    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.587396 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.587382    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.587479 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.587402    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-193.ec2.internal\": node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.599915 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.599867    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.620925 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.620901    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 00:59:57.622253 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.622231    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 00:59:57.622341 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.622268    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 00:59:57.622341 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.622322    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 00:59:57.622460 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.622381    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 00:59:57.624661 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.624638    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:57.700392 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.700358    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.722828 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.722796    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"]
Jul 14 00:59:57.722920 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.722864    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.727812 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.727795    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.727891 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.727823    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.727891 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.727834    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.729348 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.729336    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.729468 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.729452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.729529 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.729485    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.730207 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.730190    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.730283 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.730222    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.730283 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.730231    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.730283 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.730197    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.730387 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.730302    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.730387 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.730312    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.731513 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.731496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.731564 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.731530    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:59:57.732255 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.732237    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:59:57.732331 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.732271    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:59:57.732331 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.732286    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:59:57.755047 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.755027    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-193.ec2.internal\" not found" node="ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.759290 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.759273    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-193.ec2.internal\" not found" node="ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.801072 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.801046    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.821362 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.821342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9040b7d4a25b23ceee331d8a28092737-config\") pod \"kube-apiserver-proxy-ip-10-0-138-193.ec2.internal\" (UID: \"9040b7d4a25b23ceee331d8a28092737\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.821466 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.821364    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6df408c614937962b2e29e7da869d8ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal\" (UID: \"6df408c614937962b2e29e7da869d8ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.821466 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.821382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6df408c614937962b2e29e7da869d8ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal\" (UID: \"6df408c614937962b2e29e7da869d8ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.901621 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:57.901600    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:57.922070 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.922043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9040b7d4a25b23ceee331d8a28092737-config\") pod \"kube-apiserver-proxy-ip-10-0-138-193.ec2.internal\" (UID: \"9040b7d4a25b23ceee331d8a28092737\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.922151 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.922110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6df408c614937962b2e29e7da869d8ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal\" (UID: \"6df408c614937962b2e29e7da869d8ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.922151 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.922129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6df408c614937962b2e29e7da869d8ff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal\" (UID: \"6df408c614937962b2e29e7da869d8ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.922246 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.922167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6df408c614937962b2e29e7da869d8ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal\" (UID: \"6df408c614937962b2e29e7da869d8ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.922246 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.922159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9040b7d4a25b23ceee331d8a28092737-config\") pod \"kube-apiserver-proxy-ip-10-0-138-193.ec2.internal\" (UID: \"9040b7d4a25b23ceee331d8a28092737\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:57.922246 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:57.922201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6df408c614937962b2e29e7da869d8ff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal\" (UID: \"6df408c614937962b2e29e7da869d8ff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:58.002201 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:58.002180    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:58.056702 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:58.056685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:58.062087 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:58.061998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:58.102330 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:58.102304    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:58.203495 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:58.203387    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.366185 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:59.366166    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.429642 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.429616    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 00:59:59.430097 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.429757    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 00:59:59.467112 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:59.467093    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.490061 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.490033    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 00:54:58 +0000 UTC" deadline="2028-04-28 12:28:21.890853416 +0000 UTC"
Jul 14 00:59:59.490061 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.490057    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15707h28m22.400799058s"
Jul 14 00:59:59.496741 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.496725    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:59.519472 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.519423    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:59.567239 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:59.567196    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.667777 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:59.667739    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.768612 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:59.768583    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.790665 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:59.790637    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df408c614937962b2e29e7da869d8ff.slice/crio-5cd7a3f40ad8837ea69e759c098dc2272f77c1b3dedfc7bf9e820da5e18f597f WatchSource:0}: Error finding container 5cd7a3f40ad8837ea69e759c098dc2272f77c1b3dedfc7bf9e820da5e18f597f: Status 404 returned error can't find the container with id 5cd7a3f40ad8837ea69e759c098dc2272f77c1b3dedfc7bf9e820da5e18f597f
Jul 14 00:59:59.791326 ip-10-0-138-193 kubenswrapper[2578]: W0714 00:59:59.791306    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9040b7d4a25b23ceee331d8a28092737.slice/crio-b447f52e71ff1524625f6570bb98c541f661f9248b2f9f65beb0357f65758a6a WatchSource:0}: Error finding container b447f52e71ff1524625f6570bb98c541f661f9248b2f9f65beb0357f65758a6a: Status 404 returned error can't find the container with id b447f52e71ff1524625f6570bb98c541f661f9248b2f9f65beb0357f65758a6a
Jul 14 00:59:59.796171 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.796158    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:59:59.868733 ip-10-0-138-193 kubenswrapper[2578]: E0714 00:59:59.868710    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-193.ec2.internal\" not found"
Jul 14 00:59:59.939384 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.939366    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:59:59.981052 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.981033    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal"
Jul 14 00:59:59.987655 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.987639    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 00:59:59.988916 ip-10-0-138-193 kubenswrapper[2578]: I0714 00:59:59.988905    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal"
Jul 14 01:00:00.000680 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.000667    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 01:00:00.464934 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.464904    2578 apiserver.go:52] "Watching apiserver"
Jul 14 01:00:00.473605 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.473584    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 01:00:00.474015 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.473994    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kt79b","kube-system/konnectivity-agent-djfrp","openshift-cluster-node-tuning-operator/tuned-582sh","openshift-dns/node-resolver-gf759","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal","openshift-multus/multus-additional-cni-plugins-9w6t8","openshift-multus/multus-w7vdj","openshift-network-operator/iptables-alerter-nx7dk","kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs","openshift-image-registry/node-ca-z75jz","openshift-multus/network-metrics-daemon-bmkmz","openshift-network-diagnostics/network-check-target-94xh7","openshift-ovn-kubernetes/ovnkube-node-7qzjq"]
Jul 14 01:00:00.475520 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.475498    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.476861 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.476836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.478051 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 01:00:00.478051 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478029    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.478225 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478029    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 01:00:00.478225 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478032    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.478225 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478021    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-52pvx\""
Jul 14 01:00:00.478498 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.478976 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.478942    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 01:00:00.479177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.479133    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.479410 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.479394    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kzrf6\""
Jul 14 01:00:00.479815 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.479639    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.479939 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.479895    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.480495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.480476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.480602 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.480585    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.480698 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.480590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9mrb7\""
Jul 14 01:00:00.480764 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.480750    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 01:00:00.481906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.481875    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.481999 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.481878    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kmbpf\""
Jul 14 01:00:00.482227 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.482210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.483519 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.483497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.484998 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.484979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.486430 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.486412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.488922 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.488901    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.489021 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.488944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 01:00:00.489084 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.489046    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7mcvr\""
Jul 14 01:00:00.489133 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.489097    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 01:00:00.489133 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.489105    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 01:00:00.489793 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.489774    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 01:00:00.490537 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.490515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.490627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.490584    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 01:00:00.490805 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.490788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xj68r\""
Jul 14 01:00:00.490958 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.490944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 01:00:00.491543 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491518    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 01:00:00.491733 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b273d22-3594-4e4d-8024-de809ed7af04-hosts-file\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.491807 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysctl-d\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.491807 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43b22bf1-a9a1-4961-a61e-7704598d2039-tmp\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.491807 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491778    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-cni-bin\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.491807 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-system-cni-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491826    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hd67\" (UniqueName: \"kubernetes.io/projected/1b273d22-3594-4e4d-8024-de809ed7af04-kube-api-access-5hd67\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.492000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-os-release\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-hostroot\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-etc-kubernetes\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492150 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.491965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-modprobe-d\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.492150 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-lib-modules\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.492150 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-ovn\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492150 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-log-socket\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0150a597-f23b-457a-8f0a-5ded06e898a1-host\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.492336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zm9\" (UniqueName: \"kubernetes.io/projected/0150a597-f23b-457a-8f0a-5ded06e898a1-kube-api-access-88zm9\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.492336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b7867f-9334-42e0-a0cb-093e277a7f00-cni-binary-copy\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-node-log\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-cnibin\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-host\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovn-node-metrics-cert\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmmz\" (UniqueName: \"kubernetes.io/projected/43b22bf1-a9a1-4961-a61e-7704598d2039-kube-api-access-xqmmz\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovnkube-config\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.492469    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-cni-multus\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-kubernetes\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-systemd\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-kubelet\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492553    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hxlx\" (UniqueName: \"kubernetes.io/projected/86dc77e7-2874-4b0c-ba5d-5254f189030d-kube-api-access-8hxlx\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.492566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-socket-dir-parent\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-netns\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492597    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-daemon-config\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbcc7\" (UniqueName: \"kubernetes.io/projected/e1b7867f-9334-42e0-a0cb-093e277a7f00-kube-api-access-bbcc7\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-run\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-env-overrides\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-cni-bin\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovnkube-script-lib\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b273d22-3594-4e4d-8024-de809ed7af04-tmp-dir\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-run-netns\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-systemd\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-var-lib-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-k8s-cni-cncf-io\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492905    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-slash\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492953    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysctl-conf\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-sys\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.493072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.492996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-systemd-units\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-cni-netd\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0150a597-f23b-457a-8f0a-5ded06e898a1-serviceca\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-var-lib-kubelet\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-tuned\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-cni-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-multus-certs\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysconfig\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-etc-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493279    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-kubelet\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-conf-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493492    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zwvl4\""
Jul 14 01:00:00.493750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493560    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 01:00:00.494354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.493814    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.494930 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.494912    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.496382 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.496192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6rr7l\""
Jul 14 01:00:00.496382 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.496199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 01:00:00.496536 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.496424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.496536 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.496517    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:00.496642 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.496600    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 01:00:00.496892 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.496871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 01:00:00.496969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.496877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 01:00:00.497161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.497130    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 01:00:00.497231 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.497166    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tbpcp\""
Jul 14 01:00:00.497712 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.497673    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:00.497795 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.497754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:00.511533 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.511513    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cx5mf"
Jul 14 01:00:00.520777 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.520759    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cx5mf"
Jul 14 01:00:00.582410 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.582377    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 01:00:00.594422 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.594706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054c75f7-15b8-4f53-9cee-0d7dbab3952a-host-slash\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.594809 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-sys-fs\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.594809 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594764    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6jj\" (UniqueName: \"kubernetes.io/projected/ba4c10d7-0743-411a-87d5-9d9045121339-kube-api-access-xr6jj\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.594916 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovnkube-script-lib\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.594916 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447074a6-a97f-4e95-a5d4-3768f6929ba0-kubelet-config\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.595022 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-socket-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.595022 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b273d22-3594-4e4d-8024-de809ed7af04-tmp-dir\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.595022 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.594997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-run-netns\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595168 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-systemd\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595168 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-var-lib-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595168 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-k8s-cni-cncf-io\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.595168 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-slash\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595168 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysctl-conf\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-sys\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-systemd-units\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-cni-netd\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0150a597-f23b-457a-8f0a-5ded06e898a1-serviceca\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-var-lib-kubelet\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.595399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-tuned\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595407    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-cni-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-multus-certs\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5j98\" (UniqueName: \"kubernetes.io/projected/cbb2e296-cc1c-4cc8-91fb-190d20571b81-kube-api-access-g5j98\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595520    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovnkube-script-lib\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-device-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xrc\" (UniqueName: \"kubernetes.io/projected/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-kube-api-access-j8xrc\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595614    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysconfig\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-etc-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595718    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysconfig\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.595746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595720    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-kubelet\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-conf-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-kubelet\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/054c75f7-15b8-4f53-9cee-0d7dbab3952a-iptables-alerter-script\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b273d22-3594-4e4d-8024-de809ed7af04-hosts-file\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7b72\" (UniqueName: \"kubernetes.io/projected/054c75f7-15b8-4f53-9cee-0d7dbab3952a-kube-api-access-h7b72\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysctl-d\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43b22bf1-a9a1-4961-a61e-7704598d2039-tmp\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.595985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e29c106-2406-4b8b-8a35-a50c782e0cb2-konnectivity-ca\") pod \"konnectivity-agent-djfrp\" (UID: \"1e29c106-2406-4b8b-8a35-a50c782e0cb2\") " pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-cni-bin\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596044    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-system-cni-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596073    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cnibin\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b273d22-3594-4e4d-8024-de809ed7af04-tmp-dir\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e29c106-2406-4b8b-8a35-a50c782e0cb2-agent-certs\") pod \"konnectivity-agent-djfrp\" (UID: \"1e29c106-2406-4b8b-8a35-a50c782e0cb2\") " pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.596246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hd67\" (UniqueName: \"kubernetes.io/projected/1b273d22-3594-4e4d-8024-de809ed7af04-kube-api-access-5hd67\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-os-release\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596196    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-run-netns\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-systemd\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-conf-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-hostroot\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-etc-kubernetes\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596318    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-hostroot\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-modprobe-d\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-var-lib-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-lib-modules\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-ovn\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-log-socket\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0150a597-f23b-457a-8f0a-5ded06e898a1-host\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-lib-modules\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zm9\" (UniqueName: \"kubernetes.io/projected/0150a597-f23b-457a-8f0a-5ded06e898a1-kube-api-access-88zm9\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b273d22-3594-4e4d-8024-de809ed7af04-hosts-file\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447074a6-a97f-4e95-a5d4-3768f6929ba0-dbus\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.597056 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b7867f-9334-42e0-a0cb-093e277a7f00-cni-binary-copy\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596668    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-node-log\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-cnibin\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cni-binary-copy\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596797    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596829    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-host\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.596967    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-sys\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597024    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-k8s-cni-cncf-io\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysctl-d\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-slash\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovn-node-metrics-cert\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmmz\" (UniqueName: \"kubernetes.io/projected/43b22bf1-a9a1-4961-a61e-7704598d2039-kube-api-access-xqmmz\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597167    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-sysctl-conf\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovnkube-config\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.597944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-host\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-cni-multus\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-cni-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-registration-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-kubernetes\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597548    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-os-release\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597583    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-etc-openvswitch\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-systemd\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-kubelet\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597746    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-systemd\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hxlx\" (UniqueName: \"kubernetes.io/projected/86dc77e7-2874-4b0c-ba5d-5254f189030d-kube-api-access-8hxlx\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-systemd-units\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-socket-dir-parent\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597836    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-cni-netd\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.598843 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-netns\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597874    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0150a597-f23b-457a-8f0a-5ded06e898a1-serviceca\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-daemon-config\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-log-socket\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597929    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbcc7\" (UniqueName: \"kubernetes.io/projected/e1b7867f-9334-42e0-a0cb-093e277a7f00-kube-api-access-bbcc7\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0150a597-f23b-457a-8f0a-5ded06e898a1-host\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.597964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-run\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-multus-certs\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-run\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598043    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-env-overrides\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-run-ovn-kubernetes\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-cni-bin\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-cni-multus\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-system-cni-dir\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598237    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-socket-dir-parent\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovnkube-config\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-var-lib-kubelet\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-run-netns\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.599778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.598623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-modprobe-d\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-kubernetes\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1b7867f-9334-42e0-a0cb-093e277a7f00-multus-daemon-config\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-etc-kubernetes\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-node-log\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-os-release\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-cni-bin\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-host-kubelet\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599183    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/86dc77e7-2874-4b0c-ba5d-5254f189030d-run-ovn\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-system-cni-dir\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1b7867f-9334-42e0-a0cb-093e277a7f00-cni-binary-copy\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-host-var-lib-cni-bin\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1b7867f-9334-42e0-a0cb-093e277a7f00-cnibin\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.600613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.599428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/86dc77e7-2874-4b0c-ba5d-5254f189030d-env-overrides\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.601457 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.601419    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43b22bf1-a9a1-4961-a61e-7704598d2039-etc-tuned\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.601930 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.601550    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43b22bf1-a9a1-4961-a61e-7704598d2039-tmp\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.601930 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.601868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/86dc77e7-2874-4b0c-ba5d-5254f189030d-ovn-node-metrics-cert\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.606489 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.606471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zm9\" (UniqueName: \"kubernetes.io/projected/0150a597-f23b-457a-8f0a-5ded06e898a1-kube-api-access-88zm9\") pod \"node-ca-z75jz\" (UID: \"0150a597-f23b-457a-8f0a-5ded06e898a1\") " pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.606582 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.606498    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hd67\" (UniqueName: \"kubernetes.io/projected/1b273d22-3594-4e4d-8024-de809ed7af04-kube-api-access-5hd67\") pod \"node-resolver-gf759\" (UID: \"1b273d22-3594-4e4d-8024-de809ed7af04\") " pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.606776 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.606756    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbcc7\" (UniqueName: \"kubernetes.io/projected/e1b7867f-9334-42e0-a0cb-093e277a7f00-kube-api-access-bbcc7\") pod \"multus-w7vdj\" (UID: \"e1b7867f-9334-42e0-a0cb-093e277a7f00\") " pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.606940 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.606912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmmz\" (UniqueName: \"kubernetes.io/projected/43b22bf1-a9a1-4961-a61e-7704598d2039-kube-api-access-xqmmz\") pod \"tuned-582sh\" (UID: \"43b22bf1-a9a1-4961-a61e-7704598d2039\") " pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.607024 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.607003    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hxlx\" (UniqueName: \"kubernetes.io/projected/86dc77e7-2874-4b0c-ba5d-5254f189030d-kube-api-access-8hxlx\") pod \"ovnkube-node-7qzjq\" (UID: \"86dc77e7-2874-4b0c-ba5d-5254f189030d\") " pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.689630 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.689570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal" event={"ID":"9040b7d4a25b23ceee331d8a28092737","Type":"ContainerStarted","Data":"b447f52e71ff1524625f6570bb98c541f661f9248b2f9f65beb0357f65758a6a"}
Jul 14 01:00:00.690786 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.690753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal" event={"ID":"6df408c614937962b2e29e7da869d8ff","Type":"ContainerStarted","Data":"5cd7a3f40ad8837ea69e759c098dc2272f77c1b3dedfc7bf9e820da5e18f597f"}
Jul 14 01:00:00.699063 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699178 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e29c106-2406-4b8b-8a35-a50c782e0cb2-konnectivity-ca\") pod \"konnectivity-agent-djfrp\" (UID: \"1e29c106-2406-4b8b-8a35-a50c782e0cb2\") " pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.699178 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cnibin\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699178 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:00.699178 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e29c106-2406-4b8b-8a35-a50c782e0cb2-agent-certs\") pod \"konnectivity-agent-djfrp\" (UID: \"1e29c106-2406-4b8b-8a35-a50c782e0cb2\") " pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447074a6-a97f-4e95-a5d4-3768f6929ba0-dbus\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cni-binary-copy\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-registration-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699268    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-os-release\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-system-cni-dir\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699390    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054c75f7-15b8-4f53-9cee-0d7dbab3952a-host-slash\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-sys-fs\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699434    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6jj\" (UniqueName: \"kubernetes.io/projected/ba4c10d7-0743-411a-87d5-9d9045121339-kube-api-access-xr6jj\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447074a6-a97f-4e95-a5d4-3768f6929ba0-kubelet-config\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-socket-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699527    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5j98\" (UniqueName: \"kubernetes.io/projected/cbb2e296-cc1c-4cc8-91fb-190d20571b81-kube-api-access-g5j98\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-device-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699592    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xrc\" (UniqueName: \"kubernetes.io/projected/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-kube-api-access-j8xrc\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:00.699620 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/054c75f7-15b8-4f53-9cee-0d7dbab3952a-iptables-alerter-script\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7b72\" (UniqueName: \"kubernetes.io/projected/054c75f7-15b8-4f53-9cee-0d7dbab3952a-kube-api-access-h7b72\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/447074a6-a97f-4e95-a5d4-3768f6929ba0-dbus\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.699797    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.699892    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:00:01.199847559 +0000 UTC m=+3.087112182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-system-cni-dir\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-etc-selinux\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700095    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-registration-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/054c75f7-15b8-4f53-9cee-0d7dbab3952a-host-slash\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-device-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-sys-fs\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.699588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cnibin\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1e29c106-2406-4b8b-8a35-a50c782e0cb2-konnectivity-ca\") pod \"konnectivity-agent-djfrp\" (UID: \"1e29c106-2406-4b8b-8a35-a50c782e0cb2\") " pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.700618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cni-binary-copy\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/447074a6-a97f-4e95-a5d4-3768f6929ba0-kubelet-config\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:00.700618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cbb2e296-cc1c-4cc8-91fb-190d20571b81-os-release\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c10d7-0743-411a-87d5-9d9045121339-socket-dir\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.700618 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.700587    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:00.700806 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.700638    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:01.200612056 +0000 UTC m=+3.087876667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:00.700806 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700893 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cbb2e296-cc1c-4cc8-91fb-190d20571b81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.700924 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.700893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/054c75f7-15b8-4f53-9cee-0d7dbab3952a-iptables-alerter-script\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.702250 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.702226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1e29c106-2406-4b8b-8a35-a50c782e0cb2-agent-certs\") pod \"konnectivity-agent-djfrp\" (UID: \"1e29c106-2406-4b8b-8a35-a50c782e0cb2\") " pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.706745 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.706720    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:00.706745 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.706747    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:00.706901 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.706760    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:00.706901 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:00.706834    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:01.206818909 +0000 UTC m=+3.094083535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:00.709815 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.709750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7b72\" (UniqueName: \"kubernetes.io/projected/054c75f7-15b8-4f53-9cee-0d7dbab3952a-kube-api-access-h7b72\") pod \"iptables-alerter-nx7dk\" (UID: \"054c75f7-15b8-4f53-9cee-0d7dbab3952a\") " pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.710345 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.710323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6jj\" (UniqueName: \"kubernetes.io/projected/ba4c10d7-0743-411a-87d5-9d9045121339-kube-api-access-xr6jj\") pod \"aws-ebs-csi-driver-node-nsqgs\" (UID: \"ba4c10d7-0743-411a-87d5-9d9045121339\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:00.711766 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.711736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xrc\" (UniqueName: \"kubernetes.io/projected/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-kube-api-access-j8xrc\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:00.712135 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.712115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5j98\" (UniqueName: \"kubernetes.io/projected/cbb2e296-cc1c-4cc8-91fb-190d20571b81-kube-api-access-g5j98\") pod \"multus-additional-cni-plugins-9w6t8\" (UID: \"cbb2e296-cc1c-4cc8-91fb-190d20571b81\") " pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.790594 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.790508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7vdj"
Jul 14 01:00:00.797994 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.797967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z75jz"
Jul 14 01:00:00.806087 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.806055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-582sh"
Jul 14 01:00:00.812566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.812547    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gf759"
Jul 14 01:00:00.820016 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.819998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:00.820308 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.820289    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 01:00:00.826196 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.826043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9w6t8"
Jul 14 01:00:00.833451 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.833303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx7dk"
Jul 14 01:00:00.839586 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.839566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:00.844399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:00.844381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs"
Jul 14 01:00:01.031278 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.031240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb2e296_cc1c_4cc8_91fb_190d20571b81.slice/crio-e9c93d69b163313d36b4274be2e6a5c870f0d0fb15a1003a334f5b38ef50bbc8 WatchSource:0}: Error finding container e9c93d69b163313d36b4274be2e6a5c870f0d0fb15a1003a334f5b38ef50bbc8: Status 404 returned error can't find the container with id e9c93d69b163313d36b4274be2e6a5c870f0d0fb15a1003a334f5b38ef50bbc8
Jul 14 01:00:01.041424 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.041370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b22bf1_a9a1_4961_a61e_7704598d2039.slice/crio-bf627f160b312150d8833dd30e5b001535d4bdf79105eb6e4a73bf51ee34e022 WatchSource:0}: Error finding container bf627f160b312150d8833dd30e5b001535d4bdf79105eb6e4a73bf51ee34e022: Status 404 returned error can't find the container with id bf627f160b312150d8833dd30e5b001535d4bdf79105eb6e4a73bf51ee34e022
Jul 14 01:00:01.042216 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.042192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0150a597_f23b_457a_8f0a_5ded06e898a1.slice/crio-a4fcd47d61f439963697615ec34e7cb1cbbdf46d8ea1e009459d3b1cd79a0ed6 WatchSource:0}: Error finding container a4fcd47d61f439963697615ec34e7cb1cbbdf46d8ea1e009459d3b1cd79a0ed6: Status 404 returned error can't find the container with id a4fcd47d61f439963697615ec34e7cb1cbbdf46d8ea1e009459d3b1cd79a0ed6
Jul 14 01:00:01.042594 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.042545    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054c75f7_15b8_4f53_9cee_0d7dbab3952a.slice/crio-fe0cbaf8e606b8fc22e2981a9fc9e9cdd953bdd4295f92b13d4afd72b295deea WatchSource:0}: Error finding container fe0cbaf8e606b8fc22e2981a9fc9e9cdd953bdd4295f92b13d4afd72b295deea: Status 404 returned error can't find the container with id fe0cbaf8e606b8fc22e2981a9fc9e9cdd953bdd4295f92b13d4afd72b295deea
Jul 14 01:00:01.044298 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.044266    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dc77e7_2874_4b0c_ba5d_5254f189030d.slice/crio-f6ed55e96193ad56be380ad6019dda34d72a59afb5b84bdb310272f8ae1ed41d WatchSource:0}: Error finding container f6ed55e96193ad56be380ad6019dda34d72a59afb5b84bdb310272f8ae1ed41d: Status 404 returned error can't find the container with id f6ed55e96193ad56be380ad6019dda34d72a59afb5b84bdb310272f8ae1ed41d
Jul 14 01:00:01.045880 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.045816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b7867f_9334_42e0_a0cb_093e277a7f00.slice/crio-91d3635771b6e4b1fbd822391c5a48c73bafccebbf96784ad25f3fbbd6bf669c WatchSource:0}: Error finding container 91d3635771b6e4b1fbd822391c5a48c73bafccebbf96784ad25f3fbbd6bf669c: Status 404 returned error can't find the container with id 91d3635771b6e4b1fbd822391c5a48c73bafccebbf96784ad25f3fbbd6bf669c
Jul 14 01:00:01.047811 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.047778    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b273d22_3594_4e4d_8024_de809ed7af04.slice/crio-22fdbff21c97df0afaab1d98b5f5cbb6be8ba8d93542568259cb041e9e2bab89 WatchSource:0}: Error finding container 22fdbff21c97df0afaab1d98b5f5cbb6be8ba8d93542568259cb041e9e2bab89: Status 404 returned error can't find the container with id 22fdbff21c97df0afaab1d98b5f5cbb6be8ba8d93542568259cb041e9e2bab89
Jul 14 01:00:01.048164 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.048145    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4c10d7_0743_411a_87d5_9d9045121339.slice/crio-f900dccc38521589567aaa20c34c85985bd3c37de1317f3c9e3a8f29a09f4d0a WatchSource:0}: Error finding container f900dccc38521589567aaa20c34c85985bd3c37de1317f3c9e3a8f29a09f4d0a: Status 404 returned error can't find the container with id f900dccc38521589567aaa20c34c85985bd3c37de1317f3c9e3a8f29a09f4d0a
Jul 14 01:00:01.048476 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:01.048407    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e29c106_2406_4b8b_8a35_a50c782e0cb2.slice/crio-af536195e5b31951b82114e3eff312c39a7333a443c6d6737cc16d946c0eae36 WatchSource:0}: Error finding container af536195e5b31951b82114e3eff312c39a7333a443c6d6737cc16d946c0eae36: Status 404 returned error can't find the container with id af536195e5b31951b82114e3eff312c39a7333a443c6d6737cc16d946c0eae36
Jul 14 01:00:01.203373 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.203343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:01.203600 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.203401    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:01.203600 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.203510    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:01.203600 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.203533    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:01.203600 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.203582    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:02.203563134 +0000 UTC m=+4.090827739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:01.203600 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.203598    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:00:02.203591334 +0000 UTC m=+4.090855939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:01.305376 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.304347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:01.305376 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.304788    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:01.305376 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.304808    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:01.305376 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.304820    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:01.305376 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.304873    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:02.304856329 +0000 UTC m=+4.192120950 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:01.523320 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.523267    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 00:55:00 +0000 UTC" deadline="2028-02-15 15:02:24.613969427 +0000 UTC"
Jul 14 01:00:01.523320 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.523306    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13958h2m23.090667998s"
Jul 14 01:00:01.685185 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.685157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:01.685317 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:01.685279    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:01.694266 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.694238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djfrp" event={"ID":"1e29c106-2406-4b8b-8a35-a50c782e0cb2","Type":"ContainerStarted","Data":"af536195e5b31951b82114e3eff312c39a7333a443c6d6737cc16d946c0eae36"}
Jul 14 01:00:01.695537 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.695514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs" event={"ID":"ba4c10d7-0743-411a-87d5-9d9045121339","Type":"ContainerStarted","Data":"f900dccc38521589567aaa20c34c85985bd3c37de1317f3c9e3a8f29a09f4d0a"}
Jul 14 01:00:01.696798 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.696769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7vdj" event={"ID":"e1b7867f-9334-42e0-a0cb-093e277a7f00","Type":"ContainerStarted","Data":"91d3635771b6e4b1fbd822391c5a48c73bafccebbf96784ad25f3fbbd6bf669c"}
Jul 14 01:00:01.697921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.697896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx7dk" event={"ID":"054c75f7-15b8-4f53-9cee-0d7dbab3952a","Type":"ContainerStarted","Data":"fe0cbaf8e606b8fc22e2981a9fc9e9cdd953bdd4295f92b13d4afd72b295deea"}
Jul 14 01:00:01.699060 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.699040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z75jz" event={"ID":"0150a597-f23b-457a-8f0a-5ded06e898a1","Type":"ContainerStarted","Data":"a4fcd47d61f439963697615ec34e7cb1cbbdf46d8ea1e009459d3b1cd79a0ed6"}
Jul 14 01:00:01.700298 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.700258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"f6ed55e96193ad56be380ad6019dda34d72a59afb5b84bdb310272f8ae1ed41d"}
Jul 14 01:00:01.702440 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.702395    2578 generic.go:356] "Generic (PLEG): container finished" podID="6df408c614937962b2e29e7da869d8ff" containerID="7d93e57fce5295adf2240310db4b42ba76727c8731260eb3271567062064cdb4" exitCode=0
Jul 14 01:00:01.702526 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.702466    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal" event={"ID":"6df408c614937962b2e29e7da869d8ff","Type":"ContainerDied","Data":"7d93e57fce5295adf2240310db4b42ba76727c8731260eb3271567062064cdb4"}
Jul 14 01:00:01.703555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.703531    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gf759" event={"ID":"1b273d22-3594-4e4d-8024-de809ed7af04","Type":"ContainerStarted","Data":"22fdbff21c97df0afaab1d98b5f5cbb6be8ba8d93542568259cb041e9e2bab89"}
Jul 14 01:00:01.706937 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.706905    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-582sh" event={"ID":"43b22bf1-a9a1-4961-a61e-7704598d2039","Type":"ContainerStarted","Data":"bf627f160b312150d8833dd30e5b001535d4bdf79105eb6e4a73bf51ee34e022"}
Jul 14 01:00:01.711529 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:01.711504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerStarted","Data":"e9c93d69b163313d36b4274be2e6a5c870f0d0fb15a1003a334f5b38ef50bbc8"}
Jul 14 01:00:02.211506 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.211473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:02.211717 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.211534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:02.211717 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.211638    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:02.211830 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.211731    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:04.211708164 +0000 UTC m=+6.098972793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:02.212046 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.211641    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:02.212159 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.212073    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:00:04.212060755 +0000 UTC m=+6.099325374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:02.312454 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.311860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:02.312454 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.312040    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:02.312454 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.312059    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:02.312454 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.312071    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:02.312454 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.312124    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:04.312105796 +0000 UTC m=+6.199370405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:02.525347 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.525174    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 00:55:00 +0000 UTC" deadline="2027-12-11 21:36:24.666816737 +0000 UTC"
Jul 14 01:00:02.525347 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.525233    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12380h36m22.141588193s"
Jul 14 01:00:02.687462 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.686830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:02.687462 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.686976    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:02.687462 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.686830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:02.687462 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:02.687425    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:02.719011 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.718983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal" event={"ID":"9040b7d4a25b23ceee331d8a28092737","Type":"ContainerStarted","Data":"b369ad12f5ab97d9e8646bfb8f32b65a2047f235ab70d383a6c0c7bb2f96e377"}
Jul 14 01:00:02.724097 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.724063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal" event={"ID":"6df408c614937962b2e29e7da869d8ff","Type":"ContainerStarted","Data":"c987d63d90e044138a5ff32ae8ac701dcb6810a83acaac1d1c0b7abb049351cd"}
Jul 14 01:00:02.745225 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.745169    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-193.ec2.internal" podStartSLOduration=3.7451491 podStartE2EDuration="3.7451491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:02.744788139 +0000 UTC m=+4.632052768" watchObservedRunningTime="2026-07-14 01:00:02.7451491 +0000 UTC m=+4.632413728"
Jul 14 01:00:02.745564 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:02.745532    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-193.ec2.internal" podStartSLOduration=3.7455210389999998 podStartE2EDuration="3.745521039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:02.732508248 +0000 UTC m=+4.619772877" watchObservedRunningTime="2026-07-14 01:00:02.745521039 +0000 UTC m=+4.632785668"
Jul 14 01:00:03.685248 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:03.685174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:03.685734 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:03.685353    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:04.229946 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:04.229154    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:04.229946 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:04.229234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:04.229946 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.229364    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:04.229946 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.229423    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:08.229405549 +0000 UTC m=+10.116670168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:04.229946 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.229856    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:04.229946 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.229905    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:00:08.22988966 +0000 UTC m=+10.117154272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:04.330480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:04.330442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:04.330756 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.330665    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:04.330756 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.330707    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:04.330756 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.330721    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:04.330963 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.330778    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:08.330760406 +0000 UTC m=+10.218025013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:04.685618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:04.685002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:04.685618 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.685157    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:04.685618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:04.685493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:04.685618 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:04.685582    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:05.684867 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:05.684831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:05.685031 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:05.684952    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:06.684842 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:06.684799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:06.684842 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:06.684823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:06.685476 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:06.684940    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:06.685476 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:06.685365    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:07.685339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:07.685309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:07.685851 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:07.685445    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:08.264398 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:08.263582    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:08.264398 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:08.263679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:08.264398 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.263825    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:08.264398 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.263887    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:16.263867469 +0000 UTC m=+18.151132082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:08.264398 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.264311    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:08.264398 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.264363    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:00:16.264346844 +0000 UTC m=+18.151611455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:08.365356 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:08.364815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:08.365356 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.364966    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:08.365356 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.364984    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:08.365356 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.364993    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:08.365356 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.365041    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:16.365023131 +0000 UTC m=+18.252287760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:08.686332 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:08.685791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:08.686332 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.685905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:08.686332 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:08.685791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:08.686332 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:08.686298    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:09.685114 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:09.685076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:09.685306 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:09.685218    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:10.685738 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:10.685392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:10.685738 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:10.685513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:10.686338 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:10.685858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:10.686338 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:10.685975    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:11.685349 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:11.685315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:11.685521 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:11.685439    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:12.685441 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:12.685403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:12.685909 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:12.685547    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:12.685909 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:12.685607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:12.685909 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:12.685741    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:13.685075 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:13.685049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:13.685249 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:13.685174    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:14.685800 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:14.685761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:14.686220 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:14.685762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:14.686220 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:14.685904    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:14.686220 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:14.685979    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:15.684927 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:15.684892    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:15.685145 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:15.685014    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:16.325942 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:16.325905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:16.326221 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:16.325952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:16.326221 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.326030    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:16.326221 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.326043    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:16.326221 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.326088    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:32.326072348 +0000 UTC m=+34.213336953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:16.326221 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.326103    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:00:32.326096635 +0000 UTC m=+34.213361241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:16.426300 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:16.426269    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:16.426460 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.426407    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:16.426460 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.426425    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:16.426460 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.426433    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:16.426594 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.426480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:00:32.426466394 +0000 UTC m=+34.313730999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:16.685645 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:16.685566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:16.685645 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:16.685636    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:16.685900 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.685728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:16.685900 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:16.685845    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:17.684922 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:17.684886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:17.685394 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:17.685000    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:18.685614 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:18.685578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:18.686073 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:18.685630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:18.686073 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:18.685728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:18.686073 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:18.685863    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:19.685100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:19.685073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:19.685294 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:19.685182    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:20.685217 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:20.685187    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:20.685568 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:20.685390    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:20.685848 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:20.685780    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:20.685928 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:20.685900    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:20.783577 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:20.783538    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerStarted","Data":"f076ec0fc6a6a740f46b7feb6f0fbc660ced42fe0d90f3e856fabad07538fde5"}
Jul 14 01:00:20.784949 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:20.784925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs" event={"ID":"ba4c10d7-0743-411a-87d5-9d9045121339","Type":"ContainerStarted","Data":"ed01e036e842aa503dae1592ee75561cc2feb6dfe00ba2e12349d767b968bf3c"}
Jul 14 01:00:21.685626 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.685345    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:21.686184 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:21.685727    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:21.788617 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.788581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z75jz" event={"ID":"0150a597-f23b-457a-8f0a-5ded06e898a1","Type":"ContainerStarted","Data":"70228601cddc92a35de198c1bfbecb6cf35e9fe10a33e4e8f6ae84cf6ef84643"}
Jul 14 01:00:21.791255 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791235    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:00:21.791556 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791537    2578 generic.go:356] "Generic (PLEG): container finished" podID="86dc77e7-2874-4b0c-ba5d-5254f189030d" containerID="ee375ad21412f08b60dada28e3396db307e0188a76c3ecb759d5b39e5fae6c26" exitCode=1
Jul 14 01:00:21.791629 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"20fe2b8eccd7fc0ed7a2ee7bc75c253e5c00ef4fca169f027ac3a71a050e3677"}
Jul 14 01:00:21.791713 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"2470d565d4270a6d3bf03032281356dc8e07440088fc2c92900c20885198b2c1"}
Jul 14 01:00:21.791760 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"c937c576b6db1524bfa31287cd86927bd7ec395d323550242b34922851ef3383"}
Jul 14 01:00:21.791800 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"5c6a0c118e6a9e7a51bb0e7b99f217d0946d6b1ec1b5f608b03e0522a0ad6fec"}
Jul 14 01:00:21.791849 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerDied","Data":"ee375ad21412f08b60dada28e3396db307e0188a76c3ecb759d5b39e5fae6c26"}
Jul 14 01:00:21.791849 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.791837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"e3bf93daec3be8272ede70a24851b1129546869dfa36504c12f873fc89b6d7bd"}
Jul 14 01:00:21.793053 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.793028    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gf759" event={"ID":"1b273d22-3594-4e4d-8024-de809ed7af04","Type":"ContainerStarted","Data":"58d5ca2aec3fad68de47e354453ead4e32954d612a6714a4939fa2d22c5948f4"}
Jul 14 01:00:21.794392 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.794373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-582sh" event={"ID":"43b22bf1-a9a1-4961-a61e-7704598d2039","Type":"ContainerStarted","Data":"e46368718b2f26bfef2a1c050aaf464c179fbd63dc35b5e92a85d15bc5e070f0"}
Jul 14 01:00:21.795739 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.795719    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbb2e296-cc1c-4cc8-91fb-190d20571b81" containerID="f076ec0fc6a6a740f46b7feb6f0fbc660ced42fe0d90f3e856fabad07538fde5" exitCode=0
Jul 14 01:00:21.795818 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.795781    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerDied","Data":"f076ec0fc6a6a740f46b7feb6f0fbc660ced42fe0d90f3e856fabad07538fde5"}
Jul 14 01:00:21.797139 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.797109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-djfrp" event={"ID":"1e29c106-2406-4b8b-8a35-a50c782e0cb2","Type":"ContainerStarted","Data":"61a0f5357dc7b73685af62372775a529bc71e091ca4860154ee8fffad0cb7442"}
Jul 14 01:00:21.798375 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.798354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7vdj" event={"ID":"e1b7867f-9334-42e0-a0cb-093e277a7f00","Type":"ContainerStarted","Data":"b0ca09804e0641914894f3a54e3d0ca6597437788a439baf99507e3e0835c4eb"}
Jul 14 01:00:21.803011 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.802978    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z75jz" podStartSLOduration=4.248459007 podStartE2EDuration="23.802966973s" totalImagesPullingTime="19.554507966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:21.802720365 +0000 UTC m=+23.689984993" watchObservedRunningTime="2026-07-14 01:00:21.802966973 +0000 UTC m=+23.690231600"
Jul 14 01:00:21.842597 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.842554    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-582sh" podStartSLOduration=4.268254595 podStartE2EDuration="23.842540203s" totalImagesPullingTime="19.574285608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:21.842404833 +0000 UTC m=+23.729669461" watchObservedRunningTime="2026-07-14 01:00:21.842540203 +0000 UTC m=+23.729804831"
Jul 14 01:00:21.867245 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.867210    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gf759" podStartSLOduration=4.294315601 podStartE2EDuration="23.867196644s" totalImagesPullingTime="19.572881043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:21.854881144 +0000 UTC m=+23.742145771" watchObservedRunningTime="2026-07-14 01:00:21.867196644 +0000 UTC m=+23.754461249"
Jul 14 01:00:21.867722 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.867680    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-djfrp" podStartSLOduration=4.319514225 podStartE2EDuration="23.867675223s" totalImagesPullingTime="19.548160998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:21.867465993 +0000 UTC m=+23.754730620" watchObservedRunningTime="2026-07-14 01:00:21.867675223 +0000 UTC m=+23.754939851"
Jul 14 01:00:21.882358 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:21.882288    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w7vdj" podStartSLOduration=4.298167425 podStartE2EDuration="23.882276979s" totalImagesPullingTime="19.584109554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:21.881907851 +0000 UTC m=+23.769172480" watchObservedRunningTime="2026-07-14 01:00:21.882276979 +0000 UTC m=+23.769541607"
Jul 14 01:00:22.685906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:22.685867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:22.686487 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:22.685913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:22.686487 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:22.686014    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:22.686487 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:22.686111    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:22.803958 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:22.803920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx7dk" event={"ID":"054c75f7-15b8-4f53-9cee-0d7dbab3952a","Type":"ContainerStarted","Data":"aecdc22e09b14cdf816f8c0a17183e942404e8cbd6342e2fee848fe3dd9f1a05"}
Jul 14 01:00:22.818919 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:22.818875    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nx7dk" podStartSLOduration=5.266884995 podStartE2EDuration="24.818861223s" totalImagesPullingTime="19.551976228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:22.818500667 +0000 UTC m=+24.705765296" watchObservedRunningTime="2026-07-14 01:00:22.818861223 +0000 UTC m=+24.706125850"
Jul 14 01:00:22.940278 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:22.940216    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 01:00:23.539801 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.539773    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:23.540405 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.540384    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:23.556414 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.556324    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T01:00:22.940240207Z","UUID":"2b068a4e-ccaf-46d4-a74f-685d6c4c1bee","Handler":null,"Name":"","Endpoint":""}
Jul 14 01:00:23.558243 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.558217    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 01:00:23.558358 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.558261    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 01:00:23.685207 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.685172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:23.685383 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:23.685293    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:23.806853 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.806657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs" event={"ID":"ba4c10d7-0743-411a-87d5-9d9045121339","Type":"ContainerStarted","Data":"69bcdd9f1034bbd37e15325d2673e20f6c71f31f3e940d534d88eefe78fb321c"}
Jul 14 01:00:23.833480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.833450    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:23.834113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:23.833986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-djfrp"
Jul 14 01:00:24.685105 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:24.685066    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:24.685287 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:24.685184    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:24.685287 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:24.685243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:24.685409 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:24.685361    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:24.812578 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:24.812547    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:00:24.813164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:24.812980    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"450273b0aae4330b2df33819f9fb09da35e7d2b9d7d84619a1f5b39744a24806"}
Jul 14 01:00:25.684856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:25.684823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:25.685039 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:25.684950    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:25.706030 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:25.705998    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gf759_1b273d22-3594-4e4d-8024-de809ed7af04/dns-node-resolver/0.log"
Jul 14 01:00:26.685023 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.684846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:26.685596 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.684918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:26.685596 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:26.685096    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:26.685596 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:26.685176    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:26.818957 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.818925    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbb2e296-cc1c-4cc8-91fb-190d20571b81" containerID="b9326d1fb7da05f054aa2df07f18cc3ad6c0943c6a01ec11df8a19428e604d2c" exitCode=0
Jul 14 01:00:26.819092 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.819006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerDied","Data":"b9326d1fb7da05f054aa2df07f18cc3ad6c0943c6a01ec11df8a19428e604d2c"}
Jul 14 01:00:26.821181 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.821153    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs" event={"ID":"ba4c10d7-0743-411a-87d5-9d9045121339","Type":"ContainerStarted","Data":"a0d0f11279b3ee2c187f89f5e29f88b383260268385009b9d44f24bb4bd545d0"}
Jul 14 01:00:26.827575 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.827552    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:00:26.828331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.828313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"566ebde8996a5b063d8147e8701dd9a24e8725ccd3f06550f0faf2caf818022b"}
Jul 14 01:00:26.828596 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.828579    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:26.828679 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.828657    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:26.828766 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.828733    2578 scope.go:117] "RemoveContainer" containerID="ee375ad21412f08b60dada28e3396db307e0188a76c3ecb759d5b39e5fae6c26"
Jul 14 01:00:26.845169 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.845147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:26.878272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.878237    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nsqgs" podStartSLOduration=4.128832749 podStartE2EDuration="28.87822455s" totalImagesPullingTime="24.749391801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:26.877804861 +0000 UTC m=+28.765069491" watchObservedRunningTime="2026-07-14 01:00:26.87822455 +0000 UTC m=+28.765489175"
Jul 14 01:00:26.889146 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:26.889129    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z75jz_0150a597-f23b-457a-8f0a-5ded06e898a1/node-ca/0.log"
Jul 14 01:00:27.685464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:27.685428    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:27.685889 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:27.685536    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:27.834517 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:27.834493    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:00:27.834934 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:27.834911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" event={"ID":"86dc77e7-2874-4b0c-ba5d-5254f189030d","Type":"ContainerStarted","Data":"37dbbabcbee52ea543998abaabbcad5f687f04c8806a99d922d7bea09369d198"}
Jul 14 01:00:27.835316 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:27.835295    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:27.850521 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:27.850504    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:27.868963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:27.868924    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq" podStartSLOduration=9.881796263 podStartE2EDuration="29.86891222s" totalImagesPullingTime="19.987115957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:27.86762527 +0000 UTC m=+29.754889899" watchObservedRunningTime="2026-07-14 01:00:27.86891222 +0000 UTC m=+29.756176854"
Jul 14 01:00:28.686169 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:28.686136    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:28.686595 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:28.686258    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:28.686595 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:28.686308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:28.686595 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:28.686394    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:28.839174 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:28.839144    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbb2e296-cc1c-4cc8-91fb-190d20571b81" containerID="160536dd5f081c7d2a7b1a6d7ea2c08059d264fb6f7a6b586206a4f0803c1d6b" exitCode=0
Jul 14 01:00:28.839326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:28.839225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerDied","Data":"160536dd5f081c7d2a7b1a6d7ea2c08059d264fb6f7a6b586206a4f0803c1d6b"}
Jul 14 01:00:29.684904 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:29.684871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:29.685072 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:29.684968    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:30.685448 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:30.685408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:30.685839 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:30.685515    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:30.685839 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:30.685569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:30.685839 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:30.685657    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:30.845044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:30.845005    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbb2e296-cc1c-4cc8-91fb-190d20571b81" containerID="061ae8a4856dbac7d885b086e17de3a906c009a379956fa99a43f40996470673" exitCode=0
Jul 14 01:00:30.845201 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:30.845051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerDied","Data":"061ae8a4856dbac7d885b086e17de3a906c009a379956fa99a43f40996470673"}
Jul 14 01:00:31.685354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:31.685163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:31.685542 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:31.685431    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:32.348461 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:32.348424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:32.348655 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:32.348484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:32.348655 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.348596    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:32.348655 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.348602    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:32.348848 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.348670    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs podName:ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b nodeName:}" failed. No retries permitted until 2026-07-14 01:01:04.348651219 +0000 UTC m=+66.235915825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs") pod "network-metrics-daemon-bmkmz" (UID: "ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 01:00:32.348848 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.348709    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret podName:447074a6-a97f-4e95-a5d4-3768f6929ba0 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:04.34867903 +0000 UTC m=+66.235943641 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret") pod "global-pull-secret-syncer-kt79b" (UID: "447074a6-a97f-4e95-a5d4-3768f6929ba0") : object "kube-system"/"original-pull-secret" not registered
Jul 14 01:00:32.449602 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:32.449560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:32.449797 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.449739    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 01:00:32.449797 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.449759    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 01:00:32.449797 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.449771    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ckh9d for pod openshift-network-diagnostics/network-check-target-94xh7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:32.449961 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.449830    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d podName:25529fc8-8cc7-484f-889b-f9b413208193 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:04.449816174 +0000 UTC m=+66.337080780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ckh9d" (UniqueName: "kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d") pod "network-check-target-94xh7" (UID: "25529fc8-8cc7-484f-889b-f9b413208193") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 01:00:32.685427 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:32.685393    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:32.685608 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.685515    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:32.686211 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:32.685733    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:32.686211 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:32.685867    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:33.685213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:33.685183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:33.685377 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:33.685285    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:34.685847 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:34.685807    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:34.686264 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:34.685934    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:34.686322 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:34.685812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:34.686375 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:34.686354    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:35.685121 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:35.685091    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:35.685300 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:35.685183    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:35.858935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:35.858904    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbb2e296-cc1c-4cc8-91fb-190d20571b81" containerID="efc29cde333f195c0037113dd84e7869e18381d4adcb375950af42311407c2c0" exitCode=0
Jul 14 01:00:35.859355 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:35.858960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerDied","Data":"efc29cde333f195c0037113dd84e7869e18381d4adcb375950af42311407c2c0"}
Jul 14 01:00:36.685772 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:36.685744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:36.685966 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:36.685744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:36.685966 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:36.685851    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:36.685966 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:36.685911    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:36.863748 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:36.863720    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbb2e296-cc1c-4cc8-91fb-190d20571b81" containerID="049b01e7d7cdfae630752fc31e88ca2ff326ba1a9821f76301ff946569812683" exitCode=0
Jul 14 01:00:36.864090 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:36.863775    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerDied","Data":"049b01e7d7cdfae630752fc31e88ca2ff326ba1a9821f76301ff946569812683"}
Jul 14 01:00:37.685148 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:37.685116    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:37.685311 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:37.685285    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:37.871127 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:37.871097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" event={"ID":"cbb2e296-cc1c-4cc8-91fb-190d20571b81","Type":"ContainerStarted","Data":"e5b03958f95f2242ff3e8b7d1020e6813d977ccb4cbeac7464b26310dbbc886d"}
Jul 14 01:00:37.891801 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:37.891763    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9w6t8" podStartSLOduration=9.759535273000001 podStartE2EDuration="39.891748785s" totalImagesPullingTime="30.132213512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:37.890720531 +0000 UTC m=+39.777985158" watchObservedRunningTime="2026-07-14 01:00:37.891748785 +0000 UTC m=+39.779013434"
Jul 14 01:00:38.686250 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:38.686219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:38.686417 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:38.686288    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:38.686417 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:38.686307    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:38.686417 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:38.686363    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:39.685454 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:39.685427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:39.685825 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:39.685516    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:40.684774 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:40.684747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:40.684774 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:40.684766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:40.685022 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:40.684860    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:40.685022 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:40.684985    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:41.684897 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:41.684867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:41.685257 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:41.684966    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:42.685440 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:42.685410    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:42.685913 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:42.685509    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:42.685913 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:42.685575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:42.685913 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:42.685658    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:43.685344 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:43.685306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:43.685530 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:43.685414    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:44.685326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:44.685291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:44.685577 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:44.685402    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:44.685577 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:44.685468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:44.685906 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:44.685571    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:45.684913 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:45.684887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:45.685076 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:45.684984    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:46.685777 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:46.685746    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:46.686210 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:46.685902    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:46.686210 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:46.685973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:46.686210 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:46.686052    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:47.685749 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:47.685716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:47.685945 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:47.685827    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:48.686194 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:48.686163    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:48.686557 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:48.686255    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:48.686557 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:48.686284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:48.686557 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:48.686325    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:49.190320 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:49.190289    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kt79b"]
Jul 14 01:00:49.190498 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:49.190400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:49.190556 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:49.190493    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:49.193572 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:49.193532    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-94xh7"]
Jul 14 01:00:49.193738 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:49.193670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:49.193934 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:49.193788    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:49.194289 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:49.194263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bmkmz"]
Jul 14 01:00:49.194395 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:49.194350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:49.194453 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:49.194433    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:50.685594 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:50.685412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:50.686361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:50.685414    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:50.686361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:50.685412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:50.686361 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:50.685977    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:50.686361 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:50.686117    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:50.686361 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:50.686232    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:52.685523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:52.685493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:52.685982 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:52.685526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:52.685982 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:52.685501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:52.685982 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:52.685588    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94xh7" podUID="25529fc8-8cc7-484f-889b-f9b413208193"
Jul 14 01:00:52.685982 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:52.685723    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kt79b" podUID="447074a6-a97f-4e95-a5d4-3768f6929ba0"
Jul 14 01:00:52.685982 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:52.685771    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmkmz" podUID="ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b"
Jul 14 01:00:53.466391 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.466367    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-193.ec2.internal" event="NodeReady"
Jul 14 01:00:53.466536 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.466482    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 01:00:53.513045 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.512971    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zh82v"]
Jul 14 01:00:53.518214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.518199    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nstzz"]
Jul 14 01:00:53.518332 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.518314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.520467 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.520443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 01:00:53.520589 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.520569    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x8vr9\""
Jul 14 01:00:53.520727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.520710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 01:00:53.530658 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.530640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nstzz"]
Jul 14 01:00:53.530786 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.530675    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zh82v"]
Jul 14 01:00:53.530845 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.530788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.533007 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.532989    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 01:00:53.533076 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.533020    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbz48\""
Jul 14 01:00:53.534149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.533591    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 01:00:53.534149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.533654    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 01:00:53.605907 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.605881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rbx\" (UniqueName: \"kubernetes.io/projected/d9bf370b-db7b-4948-9332-ac45030fbd41-kube-api-access-84rbx\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.605907 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.605909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5a6ad2-1ee6-4761-9705-bcde606f6861-cert\") pod \"ingress-canary-nstzz\" (UID: \"4b5a6ad2-1ee6-4761-9705-bcde606f6861\") " pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.606040 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.605936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf370b-db7b-4948-9332-ac45030fbd41-config-volume\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.606040 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.605979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bf370b-db7b-4948-9332-ac45030fbd41-tmp-dir\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.606040 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.606019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9bf370b-db7b-4948-9332-ac45030fbd41-metrics-tls\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.606132 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.606038    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx4n\" (UniqueName: \"kubernetes.io/projected/4b5a6ad2-1ee6-4761-9705-bcde606f6861-kube-api-access-2tx4n\") pod \"ingress-canary-nstzz\" (UID: \"4b5a6ad2-1ee6-4761-9705-bcde606f6861\") " pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.613550 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.613529    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hlcx9"]
Jul 14 01:00:53.625542 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.625523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hlcx9"]
Jul 14 01:00:53.625628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.625606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.627977 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.627949    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 01:00:53.627977 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.627958    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 01:00:53.628104 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.628022    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 01:00:53.628273 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.628255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 01:00:53.628341 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.628275    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zmdxs\""
Jul 14 01:00:53.707069 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rbx\" (UniqueName: \"kubernetes.io/projected/d9bf370b-db7b-4948-9332-ac45030fbd41-kube-api-access-84rbx\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.707069 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5a6ad2-1ee6-4761-9705-bcde606f6861-cert\") pod \"ingress-canary-nstzz\" (UID: \"4b5a6ad2-1ee6-4761-9705-bcde606f6861\") " pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf370b-db7b-4948-9332-ac45030fbd41-config-volume\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bf370b-db7b-4948-9332-ac45030fbd41-tmp-dir\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9bf370b-db7b-4948-9332-ac45030fbd41-metrics-tls\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx4n\" (UniqueName: \"kubernetes.io/projected/4b5a6ad2-1ee6-4761-9705-bcde606f6861-kube-api-access-2tx4n\") pod \"ingress-canary-nstzz\" (UID: \"4b5a6ad2-1ee6-4761-9705-bcde606f6861\") " pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707219    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fec0415-49b6-41d8-a027-8f2ab01240ca-data-volume\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0415-49b6-41d8-a027-8f2ab01240ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpddk\" (UniqueName: \"kubernetes.io/projected/7fec0415-49b6-41d8-a027-8f2ab01240ca-kube-api-access-hpddk\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fec0415-49b6-41d8-a027-8f2ab01240ca-crio-socket\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0415-49b6-41d8-a027-8f2ab01240ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.707560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9bf370b-db7b-4948-9332-ac45030fbd41-tmp-dir\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.707887 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.707778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d9bf370b-db7b-4948-9332-ac45030fbd41-config-volume\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.711213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.711192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d9bf370b-db7b-4948-9332-ac45030fbd41-metrics-tls\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.711330 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.711313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b5a6ad2-1ee6-4761-9705-bcde606f6861-cert\") pod \"ingress-canary-nstzz\" (UID: \"4b5a6ad2-1ee6-4761-9705-bcde606f6861\") " pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.714572 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.714546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rbx\" (UniqueName: \"kubernetes.io/projected/d9bf370b-db7b-4948-9332-ac45030fbd41-kube-api-access-84rbx\") pod \"dns-default-zh82v\" (UID: \"d9bf370b-db7b-4948-9332-ac45030fbd41\") " pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.714700 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.714585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx4n\" (UniqueName: \"kubernetes.io/projected/4b5a6ad2-1ee6-4761-9705-bcde606f6861-kube-api-access-2tx4n\") pod \"ingress-canary-nstzz\" (UID: \"4b5a6ad2-1ee6-4761-9705-bcde606f6861\") " pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.808798 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.808721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fec0415-49b6-41d8-a027-8f2ab01240ca-crio-socket\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.808798 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.808760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0415-49b6-41d8-a027-8f2ab01240ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.808969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.808927    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fec0415-49b6-41d8-a027-8f2ab01240ca-crio-socket\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.808969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.808935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fec0415-49b6-41d8-a027-8f2ab01240ca-data-volume\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.809051 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.808980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0415-49b6-41d8-a027-8f2ab01240ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.809051 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.809007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpddk\" (UniqueName: \"kubernetes.io/projected/7fec0415-49b6-41d8-a027-8f2ab01240ca-kube-api-access-hpddk\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.809213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.809194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fec0415-49b6-41d8-a027-8f2ab01240ca-data-volume\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.809252 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.809200    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0415-49b6-41d8-a027-8f2ab01240ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.811272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.811253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0415-49b6-41d8-a027-8f2ab01240ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.816369 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.816349    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpddk\" (UniqueName: \"kubernetes.io/projected/7fec0415-49b6-41d8-a027-8f2ab01240ca-kube-api-access-hpddk\") pod \"insights-runtime-extractor-hlcx9\" (UID: \"7fec0415-49b6-41d8-a027-8f2ab01240ca\") " pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.828328 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.828282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:53.841934 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.841917    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nstzz"
Jul 14 01:00:53.934170 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.934069    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hlcx9"
Jul 14 01:00:53.986221 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.986183    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zh82v"]
Jul 14 01:00:53.989350 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:53.989327    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nstzz"]
Jul 14 01:00:53.991286 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:53.991233    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bf370b_db7b_4948_9332_ac45030fbd41.slice/crio-d0976dc07b7748e778ae567a2d939b81d62ca7a2bfe30802c464fa86875f7346 WatchSource:0}: Error finding container d0976dc07b7748e778ae567a2d939b81d62ca7a2bfe30802c464fa86875f7346: Status 404 returned error can't find the container with id d0976dc07b7748e778ae567a2d939b81d62ca7a2bfe30802c464fa86875f7346
Jul 14 01:00:53.997271 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:53.997238    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5a6ad2_1ee6_4761_9705_bcde606f6861.slice/crio-1bf94d440cc4ba28bf41a9ade601e58c6b851a0c6305b4f685215462fe41159e WatchSource:0}: Error finding container 1bf94d440cc4ba28bf41a9ade601e58c6b851a0c6305b4f685215462fe41159e: Status 404 returned error can't find the container with id 1bf94d440cc4ba28bf41a9ade601e58c6b851a0c6305b4f685215462fe41159e
Jul 14 01:00:54.077240 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.077164    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hlcx9"]
Jul 14 01:00:54.080929 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:54.080904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fec0415_49b6_41d8_a027_8f2ab01240ca.slice/crio-63b2cf2cb564cd4db54d337f02c706b2d5f32e246b6a2017692c00d2abadafc5 WatchSource:0}: Error finding container 63b2cf2cb564cd4db54d337f02c706b2d5f32e246b6a2017692c00d2abadafc5: Status 404 returned error can't find the container with id 63b2cf2cb564cd4db54d337f02c706b2d5f32e246b6a2017692c00d2abadafc5
Jul 14 01:00:54.689869 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.688202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:00:54.689869 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.688722    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:00:54.689869 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.689198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:00:54.692664 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.691296    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 01:00:54.692664 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.691716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 01:00:54.692874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.692851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 01:00:54.692874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.692861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-btv6j\""
Jul 14 01:00:54.692986 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.692917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hm62n\""
Jul 14 01:00:54.693074 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.693060    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 01:00:54.909727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.909669    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nstzz" event={"ID":"4b5a6ad2-1ee6-4761-9705-bcde606f6861","Type":"ContainerStarted","Data":"1bf94d440cc4ba28bf41a9ade601e58c6b851a0c6305b4f685215462fe41159e"}
Jul 14 01:00:54.912215 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.912151    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh82v" event={"ID":"d9bf370b-db7b-4948-9332-ac45030fbd41","Type":"ContainerStarted","Data":"d0976dc07b7748e778ae567a2d939b81d62ca7a2bfe30802c464fa86875f7346"}
Jul 14 01:00:54.914308 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:54.914248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlcx9" event={"ID":"7fec0415-49b6-41d8-a027-8f2ab01240ca","Type":"ContainerStarted","Data":"63b2cf2cb564cd4db54d337f02c706b2d5f32e246b6a2017692c00d2abadafc5"}
Jul 14 01:00:56.908815 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.908781    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c54499bcb-k4sws"]
Jul 14 01:00:56.931432 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.931390    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c54499bcb-k4sws"]
Jul 14 01:00:56.931588 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.931509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:56.933812 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.933782    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 14 01:00:56.934097 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934081    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 14 01:00:56.935164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934717    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 14 01:00:56.935164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934739    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 14 01:00:56.935164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934791    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 14 01:00:56.935164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934729    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xt9b8\""
Jul 14 01:00:56.935164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 14 01:00:56.935164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.934792    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 14 01:00:56.938783 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:56.938764    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 14 01:00:57.034099 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-trusted-ca-bundle\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.034284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-service-ca\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.034284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6tb\" (UniqueName: \"kubernetes.io/projected/93f20eb6-86b6-41c1-b171-b3e64238922f-kube-api-access-hc6tb\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.034284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-oauth-config\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.034284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-serving-cert\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.034284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-oauth-serving-cert\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.034494 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.034335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-console-config\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.134935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.134896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-oauth-serving-cert\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.134972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-console-config\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135018    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-trusted-ca-bundle\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-service-ca\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6tb\" (UniqueName: \"kubernetes.io/projected/93f20eb6-86b6-41c1-b171-b3e64238922f-kube-api-access-hc6tb\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135096    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-oauth-config\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135269 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135114    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-serving-cert\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135599 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-oauth-serving-cert\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.135998 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-service-ca\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.136108 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.135978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-console-config\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.136249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.136226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-trusted-ca-bundle\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.140080 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.140055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-serving-cert\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.140237 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.140212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-oauth-config\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.143655 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.143632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6tb\" (UniqueName: \"kubernetes.io/projected/93f20eb6-86b6-41c1-b171-b3e64238922f-kube-api-access-hc6tb\") pod \"console-6c54499bcb-k4sws\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") " pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:57.255429 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:57.255388    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:00:58.055427 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.055385    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hfzz9"]
Jul 14 01:00:58.064420 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.064399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.066861 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.066834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hfzz9"]
Jul 14 01:00:58.067341 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.067316    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 01:00:58.067823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.067631    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 01:00:58.067823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.067653    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-f6gj2\""
Jul 14 01:00:58.067823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.067712    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 01:00:58.067823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.067800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 01:00:58.068050 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.068005    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 01:00:58.143209 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.143172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.143383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.143313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92fd7c8-fb30-40bd-8697-844227edef4f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.143383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.143345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzzxh\" (UniqueName: \"kubernetes.io/projected/c92fd7c8-fb30-40bd-8697-844227edef4f-kube-api-access-nzzxh\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.143505 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.143442    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.244552 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.244518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.244788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.244608    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92fd7c8-fb30-40bd-8697-844227edef4f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.244788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.244631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzzxh\" (UniqueName: \"kubernetes.io/projected/c92fd7c8-fb30-40bd-8697-844227edef4f-kube-api-access-nzzxh\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.244788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.244703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.244981 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:58.244802    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 14 01:00:58.244981 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:00:58.244854    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-tls podName:c92fd7c8-fb30-40bd-8697-844227edef4f nodeName:}" failed. No retries permitted until 2026-07-14 01:00:58.744836464 +0000 UTC m=+60.632101079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-tls") pod "prometheus-operator-64785b6675-hfzz9" (UID: "c92fd7c8-fb30-40bd-8697-844227edef4f") : secret "prometheus-operator-tls" not found
Jul 14 01:00:58.246108 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.246075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c92fd7c8-fb30-40bd-8697-844227edef4f-metrics-client-ca\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.253632 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.253350    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.256196 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.256147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzzxh\" (UniqueName: \"kubernetes.io/projected/c92fd7c8-fb30-40bd-8697-844227edef4f-kube-api-access-nzzxh\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.750172 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.750137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.752646 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.752620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c92fd7c8-fb30-40bd-8697-844227edef4f-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-hfzz9\" (UID: \"c92fd7c8-fb30-40bd-8697-844227edef4f\") " pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:58.978594 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.978526    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-f6gj2\""
Jul 14 01:00:58.986315 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:58.986289    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9"
Jul 14 01:00:59.743167 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.742112    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c54499bcb-k4sws"]
Jul 14 01:00:59.757538 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.757412    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-hfzz9"]
Jul 14 01:00:59.762026 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:00:59.761992    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92fd7c8_fb30_40bd_8697_844227edef4f.slice/crio-7e1de3adff8b251d6cf7a133b84366abeb88eb175c104362556eaf0dbe8bf664 WatchSource:0}: Error finding container 7e1de3adff8b251d6cf7a133b84366abeb88eb175c104362556eaf0dbe8bf664: Status 404 returned error can't find the container with id 7e1de3adff8b251d6cf7a133b84366abeb88eb175c104362556eaf0dbe8bf664
Jul 14 01:00:59.852814 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.852738    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7qzjq"
Jul 14 01:00:59.928848 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.928814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlcx9" event={"ID":"7fec0415-49b6-41d8-a027-8f2ab01240ca","Type":"ContainerStarted","Data":"101eaf219fcf96498ba7fc953e10d55361c4b004ae1038c79bfb727ae595da00"}
Jul 14 01:00:59.929862 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.929830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9" event={"ID":"c92fd7c8-fb30-40bd-8697-844227edef4f","Type":"ContainerStarted","Data":"7e1de3adff8b251d6cf7a133b84366abeb88eb175c104362556eaf0dbe8bf664"}
Jul 14 01:00:59.931183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.931159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nstzz" event={"ID":"4b5a6ad2-1ee6-4761-9705-bcde606f6861","Type":"ContainerStarted","Data":"38859c6e7897282857520828d224f6005a97e2013691853b3bab99d5006ee6f5"}
Jul 14 01:00:59.932829 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.932798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh82v" event={"ID":"d9bf370b-db7b-4948-9332-ac45030fbd41","Type":"ContainerStarted","Data":"2e83ad5250d61a007704bb09ff3d7d9e2b1463e16c9647882393148e237a9c42"}
Jul 14 01:00:59.932953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.932842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zh82v" event={"ID":"d9bf370b-db7b-4948-9332-ac45030fbd41","Type":"ContainerStarted","Data":"89366a801f68f8e40c571886bb3c2cd161b032b033c7d44d5f846d6c1841ae11"}
Jul 14 01:00:59.932953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.932872    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zh82v"
Jul 14 01:00:59.933864 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.933840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c54499bcb-k4sws" event={"ID":"93f20eb6-86b6-41c1-b171-b3e64238922f","Type":"ContainerStarted","Data":"40b6de7b0837ffb8998b4c80507f3eb5c4ea6c007d1e8e11d3a7389fc93e5750"}
Jul 14 01:00:59.946701 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:00:59.946648    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nstzz" podStartSLOduration=1.3657401789999999 podStartE2EDuration="6.946636287s" totalImagesPullingTime="5.580896108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:59.946013065 +0000 UTC m=+61.833277694" watchObservedRunningTime="2026-07-14 01:00:59.946636287 +0000 UTC m=+61.833900915"
Jul 14 01:01:01.942156 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:01.942120    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlcx9" event={"ID":"7fec0415-49b6-41d8-a027-8f2ab01240ca","Type":"ContainerStarted","Data":"25823a26d9d1a942705a62d6098bf8b8e4dfd1730d4848a37fc551ced3badcaf"}
Jul 14 01:01:03.961754 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:03.961710    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c54499bcb-k4sws" event={"ID":"93f20eb6-86b6-41c1-b171-b3e64238922f","Type":"ContainerStarted","Data":"227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c"}
Jul 14 01:01:03.963703 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:03.963652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9" event={"ID":"c92fd7c8-fb30-40bd-8697-844227edef4f","Type":"ContainerStarted","Data":"36cf02ea78e20bdd8d0265a013a5525c8caac42752e7e7d96e9e502c2240d791"}
Jul 14 01:01:03.963832 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:03.963726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9" event={"ID":"c92fd7c8-fb30-40bd-8697-844227edef4f","Type":"ContainerStarted","Data":"063af48e38cdca0ffeedd0cbeee2bee757e922d658e9335b46522525d64d497a"}
Jul 14 01:01:03.978119 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:03.978075    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c54499bcb-k4sws" podStartSLOduration=4.679337952 podStartE2EDuration="7.97806182s" totalImagesPullingTime="3.298723868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:03.977110401 +0000 UTC m=+65.864375031" watchObservedRunningTime="2026-07-14 01:01:03.97806182 +0000 UTC m=+65.865326448"
Jul 14 01:01:03.978457 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:03.978436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zh82v" podStartSLOduration=5.717617606 podStartE2EDuration="10.978428493s" totalImagesPullingTime="5.260810887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:00:59.974989188 +0000 UTC m=+61.862253817" watchObservedRunningTime="2026-07-14 01:01:03.978428493 +0000 UTC m=+65.865693178"
Jul 14 01:01:03.995348 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:03.995298    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-hfzz9" podStartSLOduration=2.712340561 podStartE2EDuration="5.995280498s" totalImagesPullingTime="3.282939937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:03.994783377 +0000 UTC m=+65.882048005" watchObservedRunningTime="2026-07-14 01:01:03.995280498 +0000 UTC m=+65.882545126"
Jul 14 01:01:04.399357 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.399321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:01:04.399531 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.399394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:01:04.401732 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.401706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 01:01:04.402319 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.402300    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 01:01:04.412846 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.412821    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/447074a6-a97f-4e95-a5d4-3768f6929ba0-original-pull-secret\") pod \"global-pull-secret-syncer-kt79b\" (UID: \"447074a6-a97f-4e95-a5d4-3768f6929ba0\") " pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:01:04.413503 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.413483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b-metrics-certs\") pod \"network-metrics-daemon-bmkmz\" (UID: \"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b\") " pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:01:04.499941 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.499895    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:01:04.502350 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.502325    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 01:01:04.512240 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.512212    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 01:01:04.523938 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.523882    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckh9d\" (UniqueName: \"kubernetes.io/projected/25529fc8-8cc7-484f-889b-f9b413208193-kube-api-access-ckh9d\") pod \"network-check-target-94xh7\" (UID: \"25529fc8-8cc7-484f-889b-f9b413208193\") " pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:01:04.609162 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.609130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hm62n\""
Jul 14 01:01:04.617889 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.617864    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmkmz"
Jul 14 01:01:04.632901 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.632880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-btv6j\""
Jul 14 01:01:04.641602 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.641583    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:01:04.660853 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.660788    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kt79b"
Jul 14 01:01:04.935775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.935744    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bmkmz"]
Jul 14 01:01:04.939791 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:04.939763    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7999a0_5121_4f08_9ea0_0b2ed0d7b88b.slice/crio-44a97a09120f983c38b65569ea0a7282d3d55ec32dd0b47d35be9ee46285f6a3 WatchSource:0}: Error finding container 44a97a09120f983c38b65569ea0a7282d3d55ec32dd0b47d35be9ee46285f6a3: Status 404 returned error can't find the container with id 44a97a09120f983c38b65569ea0a7282d3d55ec32dd0b47d35be9ee46285f6a3
Jul 14 01:01:04.971227 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.971187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hlcx9" event={"ID":"7fec0415-49b6-41d8-a027-8f2ab01240ca","Type":"ContainerStarted","Data":"2ea2578ff346484bed5e89d7e19c38f4b6994413143a1e239bb1ca3cbebca80b"}
Jul 14 01:01:04.972377 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.972349    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bmkmz" event={"ID":"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b","Type":"ContainerStarted","Data":"44a97a09120f983c38b65569ea0a7282d3d55ec32dd0b47d35be9ee46285f6a3"}
Jul 14 01:01:04.987330 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:04.987291    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hlcx9" podStartSLOduration=1.408744442 podStartE2EDuration="11.987278451s" totalImagesPullingTime="10.578534009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 01:00:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:04.986802359 +0000 UTC m=+66.874066987" watchObservedRunningTime="2026-07-14 01:01:04.987278451 +0000 UTC m=+66.874543078"
Jul 14 01:01:05.159612 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.159578    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-94xh7"]
Jul 14 01:01:05.163376 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.163351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kt79b"]
Jul 14 01:01:05.163943 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:05.163916    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25529fc8_8cc7_484f_889b_f9b413208193.slice/crio-4d2e6b5f5234cc016280d6a396bc921a71e6796382289eb3b9ad027259e60cf9 WatchSource:0}: Error finding container 4d2e6b5f5234cc016280d6a396bc921a71e6796382289eb3b9ad027259e60cf9: Status 404 returned error can't find the container with id 4d2e6b5f5234cc016280d6a396bc921a71e6796382289eb3b9ad027259e60cf9
Jul 14 01:01:05.166407 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:05.166388    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447074a6_a97f_4e95_a5d4_3768f6929ba0.slice/crio-92b8016ac9fdc6aaa5962a6441abd3f63506b7aece84a2ab49db15244e2c7d44 WatchSource:0}: Error finding container 92b8016ac9fdc6aaa5962a6441abd3f63506b7aece84a2ab49db15244e2c7d44: Status 404 returned error can't find the container with id 92b8016ac9fdc6aaa5962a6441abd3f63506b7aece84a2ab49db15244e2c7d44
Jul 14 01:01:05.386656 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.386627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"]
Jul 14 01:01:05.389780 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.389761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.392130 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.392110    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-558p9\""
Jul 14 01:01:05.392391 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.392373    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 01:01:05.392538 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.392524    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 01:01:05.399510 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.399488    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"]
Jul 14 01:01:05.412615 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.412587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"]
Jul 14 01:01:05.419909 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.419883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jqlfq"]
Jul 14 01:01:05.420075 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.420057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.422184 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.422161    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r6lhb\""
Jul 14 01:01:05.422267 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.422247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.422582 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.422536    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 01:01:05.422735 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.422611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 01:01:05.422735 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.422652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 01:01:05.424899 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.424577    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 01:01:05.424899 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.424585    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 01:01:05.425719 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.425678    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fhkh5\""
Jul 14 01:01:05.425922 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.425907    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 01:01:05.426140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.426127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"]
Jul 14 01:01:05.509484 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp8xg\" (UniqueName: \"kubernetes.io/projected/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-kube-api-access-gp8xg\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.509484 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.509661 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-wtmp\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.509661 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.509661 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.509661 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509651    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44919429-543c-4426-9b31-bff4ed86fd96-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509670    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmdb\" (UniqueName: \"kubernetes.io/projected/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-api-access-ddmdb\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6n6\" (UniqueName: \"kubernetes.io/projected/44919429-543c-4426-9b31-bff4ed86fd96-kube-api-access-qq6n6\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-root\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.509838 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44919429-543c-4426-9b31-bff4ed86fd96-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.510089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-sys\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.510089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509861    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-textfile\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.510089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-tls\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.510089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44919429-543c-4426-9b31-bff4ed86fd96-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.510089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.509976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-metrics-client-ca\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.610654 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.610631    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.610804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.610659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44919429-543c-4426-9b31-bff4ed86fd96-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.610804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.610679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmdb\" (UniqueName: \"kubernetes.io/projected/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-api-access-ddmdb\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.610804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.610724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6n6\" (UniqueName: \"kubernetes.io/projected/44919429-543c-4426-9b31-bff4ed86fd96-kube-api-access-qq6n6\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.610953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.610912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611006 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.610974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.611061 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611008    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.611061 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-root\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611151 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.611151 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44919429-543c-4426-9b31-bff4ed86fd96-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.611151 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-sys\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-textfile\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-tls\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44919429-543c-4426-9b31-bff4ed86fd96-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.611295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-metrics-client-ca\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp8xg\" (UniqueName: \"kubernetes.io/projected/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-kube-api-access-gp8xg\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.611523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-wtmp\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611360    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.611523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/44919429-543c-4426-9b31-bff4ed86fd96-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.611741 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:01:05.611559    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 14 01:01:05.611741 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:01:05.611615    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-tls podName:8c0e0edb-e690-47fe-b35c-c8ef1da44b49 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:06.111595576 +0000 UTC m=+67.998860196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-tls") pod "node-exporter-jqlfq" (UID: "8c0e0edb-e690-47fe-b35c-c8ef1da44b49") : secret "node-exporter-tls" not found
Jul 14 01:01:05.612333 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-sys\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.612333 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.611987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-accelerators-collector-config\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.612333 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:01:05.612097    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 14 01:01:05.612333 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.612201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-textfile\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.612333 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.612272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-root\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.612333 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:01:05.612312    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-tls podName:c741dbc7-4e9c-4677-8f5c-ba80b9804ed7 nodeName:}" failed. No retries permitted until 2026-07-14 01:01:06.112282174 +0000 UTC m=+67.999546782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-4kjk2" (UID: "c741dbc7-4e9c-4677-8f5c-ba80b9804ed7") : secret "kube-state-metrics-tls" not found
Jul 14 01:01:05.612798 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.612405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-wtmp\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.612912 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.612890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-metrics-client-ca\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.613204 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.613176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.613647 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.613563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.614449 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.613848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.614449 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.614150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.614449 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.614301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.614804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.614783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/44919429-543c-4426-9b31-bff4ed86fd96-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.614926 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.614901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/44919429-543c-4426-9b31-bff4ed86fd96-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.618659 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.618623    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6n6\" (UniqueName: \"kubernetes.io/projected/44919429-543c-4426-9b31-bff4ed86fd96-kube-api-access-qq6n6\") pod \"openshift-state-metrics-84bd68b896-pn9vp\" (UID: \"44919429-543c-4426-9b31-bff4ed86fd96\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.619917 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.619896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmdb\" (UniqueName: \"kubernetes.io/projected/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-api-access-ddmdb\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:05.620058 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.620032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp8xg\" (UniqueName: \"kubernetes.io/projected/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-kube-api-access-gp8xg\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:05.698901 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.698862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"
Jul 14 01:01:05.844499 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.844466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp"]
Jul 14 01:01:05.848147 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:05.848118    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44919429_543c_4426_9b31_bff4ed86fd96.slice/crio-d6229697090be693e90bb4e842a727e86a613b20392d7465f20b68ebcbaa9813 WatchSource:0}: Error finding container d6229697090be693e90bb4e842a727e86a613b20392d7465f20b68ebcbaa9813: Status 404 returned error can't find the container with id d6229697090be693e90bb4e842a727e86a613b20392d7465f20b68ebcbaa9813
Jul 14 01:01:05.977207 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.977132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kt79b" event={"ID":"447074a6-a97f-4e95-a5d4-3768f6929ba0","Type":"ContainerStarted","Data":"92b8016ac9fdc6aaa5962a6441abd3f63506b7aece84a2ab49db15244e2c7d44"}
Jul 14 01:01:05.978741 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.978713    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-94xh7" event={"ID":"25529fc8-8cc7-484f-889b-f9b413208193","Type":"ContainerStarted","Data":"4d2e6b5f5234cc016280d6a396bc921a71e6796382289eb3b9ad027259e60cf9"}
Jul 14 01:01:05.980976 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.980949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp" event={"ID":"44919429-543c-4426-9b31-bff4ed86fd96","Type":"ContainerStarted","Data":"00dd026167a928ecd78769cc3743c8d9920eb41c24f153bcdfd8317805b4eb9a"}
Jul 14 01:01:05.981090 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:05.981006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp" event={"ID":"44919429-543c-4426-9b31-bff4ed86fd96","Type":"ContainerStarted","Data":"d6229697090be693e90bb4e842a727e86a613b20392d7465f20b68ebcbaa9813"}
Jul 14 01:01:06.115550 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.115508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-tls\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:06.115723 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.115664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:06.117924 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.117898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c0e0edb-e690-47fe-b35c-c8ef1da44b49-node-exporter-tls\") pod \"node-exporter-jqlfq\" (UID: \"8c0e0edb-e690-47fe-b35c-c8ef1da44b49\") " pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:06.118065 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.118051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c741dbc7-4e9c-4677-8f5c-ba80b9804ed7-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-4kjk2\" (UID: \"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:06.331672 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.331584    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"
Jul 14 01:01:06.337073 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.337052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jqlfq"
Jul 14 01:01:06.347549 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:06.347509    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0e0edb_e690_47fe_b35c_c8ef1da44b49.slice/crio-e0e43a955fe7774df9b641eb12c8b966488fee7d3954166535c925dd1e215eb1 WatchSource:0}: Error finding container e0e43a955fe7774df9b641eb12c8b966488fee7d3954166535c925dd1e215eb1: Status 404 returned error can't find the container with id e0e43a955fe7774df9b641eb12c8b966488fee7d3954166535c925dd1e215eb1
Jul 14 01:01:06.482232 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.482199    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 01:01:06.488468 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.488443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.489073 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.489044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-4kjk2"]
Jul 14 01:01:06.491077 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.490934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 14 01:01:06.491077 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.490962    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 14 01:01:06.491077 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.490985    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gxkcs\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491313    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491353    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491449    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491530    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491651    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 14 01:01:06.491785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.491663    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 14 01:01:06.492989 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:06.492969    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc741dbc7_4e9c_4677_8f5c_ba80b9804ed7.slice/crio-e3676bea1079d2564265c83be2c20c9ce565027e06e1602a0c320090c99ab19c WatchSource:0}: Error finding container e3676bea1079d2564265c83be2c20c9ce565027e06e1602a0c320090c99ab19c: Status 404 returned error can't find the container with id e3676bea1079d2564265c83be2c20c9ce565027e06e1602a0c320090c99ab19c
Jul 14 01:01:06.502482 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.502460    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 01:01:06.620297 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620135    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620297 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620297 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620203    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620323    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-config-out\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620483    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rxc\" (UniqueName: \"kubernetes.io/projected/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-kube-api-access-76rxc\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620873 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-web-config\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.620873 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.620585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721350 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-web-config\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721521 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721521 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721521 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721521 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721572    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-config-out\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721605    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.721746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.722069 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.721760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rxc\" (UniqueName: \"kubernetes.io/projected/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-kube-api-access-76rxc\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.722640 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.722608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.723351 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.723066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.723788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.723722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.724795 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.724748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-web-config\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.725647 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.725621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.727003 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.726980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-config-out\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.727155 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.727123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-config-volume\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.727523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.727481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.727980 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.727940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.727980 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.727971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.728442 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.728415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.729277 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.729230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.730189 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.730165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rxc\" (UniqueName: \"kubernetes.io/projected/a5909f41-52a5-4a5b-8220-f00bd9c71fe5-kube-api-access-76rxc\") pod \"alertmanager-main-0\" (UID: \"a5909f41-52a5-4a5b-8220-f00bd9c71fe5\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.807813 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.807780    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 01:01:06.975644 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.975609    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 01:01:06.979084 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:06.979057    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5909f41_52a5_4a5b_8220_f00bd9c71fe5.slice/crio-dfa826617edcb5a953453cf8d009748b9b7740b0fd945ef2a48563cbd6d280f8 WatchSource:0}: Error finding container dfa826617edcb5a953453cf8d009748b9b7740b0fd945ef2a48563cbd6d280f8: Status 404 returned error can't find the container with id dfa826617edcb5a953453cf8d009748b9b7740b0fd945ef2a48563cbd6d280f8
Jul 14 01:01:06.987309 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.987278    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp" event={"ID":"44919429-543c-4426-9b31-bff4ed86fd96","Type":"ContainerStarted","Data":"0ec12d234046124bd03c696f5a08b73fcea6fdfb7b0252c17ec2eeaaa192753d"}
Jul 14 01:01:06.988813 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.988730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"dfa826617edcb5a953453cf8d009748b9b7740b0fd945ef2a48563cbd6d280f8"}
Jul 14 01:01:06.990276 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.990220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqlfq" event={"ID":"8c0e0edb-e690-47fe-b35c-c8ef1da44b49","Type":"ContainerStarted","Data":"e0e43a955fe7774df9b641eb12c8b966488fee7d3954166535c925dd1e215eb1"}
Jul 14 01:01:06.991516 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:06.991486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2" event={"ID":"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7","Type":"ContainerStarted","Data":"e3676bea1079d2564265c83be2c20c9ce565027e06e1602a0c320090c99ab19c"}
Jul 14 01:01:07.256432 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:07.255787    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:01:07.256432 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:07.255857    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:01:07.262701 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:07.262415    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:01:08.000466 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.000398    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:01:08.489343 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.489313    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84c4456fc9-rh78x"]
Jul 14 01:01:08.507702 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.507656    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84c4456fc9-rh78x"]
Jul 14 01:01:08.507941 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.507911    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.510364 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510340    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 14 01:01:08.510541 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510527    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 14 01:01:08.510672 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 14 01:01:08.511140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510868    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 14 01:01:08.511140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6ld95\""
Jul 14 01:01:08.511140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4cqn2301hl01c\""
Jul 14 01:01:08.511140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.510884    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 14 01:01:08.637736 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.637736 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.637929 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-grpc-tls\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.637929 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-tls\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.637929 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.637929 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.637929 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011dd403-949d-4bea-8f16-c3d6e45ef275-metrics-client-ca\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.638076 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.637937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgld\" (UniqueName: \"kubernetes.io/projected/011dd403-949d-4bea-8f16-c3d6e45ef275-kube-api-access-gqgld\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-grpc-tls\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-tls\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739458    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739486    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011dd403-949d-4bea-8f16-c3d6e45ef275-metrics-client-ca\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.739628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.739501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgld\" (UniqueName: \"kubernetes.io/projected/011dd403-949d-4bea-8f16-c3d6e45ef275-kube-api-access-gqgld\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.740847 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.740814    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/011dd403-949d-4bea-8f16-c3d6e45ef275-metrics-client-ca\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.742789 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.742742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-tls\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.742886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.742824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.742954 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.742933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.743476 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.743435    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-grpc-tls\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.743752 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.743724    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.746147 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.746120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/011dd403-949d-4bea-8f16-c3d6e45ef275-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.747975 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.747954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgld\" (UniqueName: \"kubernetes.io/projected/011dd403-949d-4bea-8f16-c3d6e45ef275-kube-api-access-gqgld\") pod \"thanos-querier-84c4456fc9-rh78x\" (UID: \"011dd403-949d-4bea-8f16-c3d6e45ef275\") " pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:08.820901 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:08.820856    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:09.695923 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.695888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-645c9f5b98-p4wr9"]
Jul 14 01:01:09.710960 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.710933    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-645c9f5b98-p4wr9"]
Jul 14 01:01:09.711149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.711043    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.713410 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.713255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fpjx2\""
Jul 14 01:01:09.713410 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.713258    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 14 01:01:09.713626 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.713611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 14 01:01:09.713734 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.713649    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 01:01:09.714214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.714194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7l5evcfpm72vv\""
Jul 14 01:01:09.714214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.714206    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 14 01:01:09.847785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.847749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-secret-metrics-server-tls\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.847966 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.847798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-secret-metrics-server-client-certs\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.847966 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.847827    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b4679efc-70e8-4162-a5b2-bf783070fbed-audit-log\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.847966 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.847866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwhp8\" (UniqueName: \"kubernetes.io/projected/b4679efc-70e8-4162-a5b2-bf783070fbed-kube-api-access-fwhp8\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.847966 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.847895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-client-ca-bundle\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.848186 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.847991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b4679efc-70e8-4162-a5b2-bf783070fbed-metrics-server-audit-profiles\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.848186 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.848026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4679efc-70e8-4162-a5b2-bf783070fbed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.938784 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.938749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zh82v"
Jul 14 01:01:09.949407 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwhp8\" (UniqueName: \"kubernetes.io/projected/b4679efc-70e8-4162-a5b2-bf783070fbed-kube-api-access-fwhp8\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949407 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949383    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-client-ca-bundle\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b4679efc-70e8-4162-a5b2-bf783070fbed-metrics-server-audit-profiles\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4679efc-70e8-4162-a5b2-bf783070fbed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949537    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-secret-metrics-server-tls\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-secret-metrics-server-client-certs\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b4679efc-70e8-4162-a5b2-bf783070fbed-audit-log\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.949984 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.949961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b4679efc-70e8-4162-a5b2-bf783070fbed-audit-log\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.950369 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.950325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4679efc-70e8-4162-a5b2-bf783070fbed-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.950753 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.950730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b4679efc-70e8-4162-a5b2-bf783070fbed-metrics-server-audit-profiles\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.952806 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.952764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-secret-metrics-server-tls\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.952923 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.952905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-secret-metrics-server-client-certs\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.953227 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.953203    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b4679efc-70e8-4162-a5b2-bf783070fbed-client-ca-bundle\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:09.959539 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:09.959516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwhp8\" (UniqueName: \"kubernetes.io/projected/b4679efc-70e8-4162-a5b2-bf783070fbed-kube-api-access-fwhp8\") pod \"metrics-server-645c9f5b98-p4wr9\" (UID: \"b4679efc-70e8-4162-a5b2-bf783070fbed\") " pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:10.022718 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.022674    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:10.186994 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.186962    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"]
Jul 14 01:01:10.190107 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.190082    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.192299 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.192274    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 14 01:01:10.192385 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.192277    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2nfjq\""
Jul 14 01:01:10.198116 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.198053    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"]
Jul 14 01:01:10.253221 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.253142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqv5t\" (UniqueName: \"kubernetes.io/projected/307fb8b3-6ab3-44b4-8ac1-3253e552749b-kube-api-access-cqv5t\") pod \"monitoring-plugin-5f68df7f58-dsczj\" (UID: \"307fb8b3-6ab3-44b4-8ac1-3253e552749b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.253352 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.253281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/307fb8b3-6ab3-44b4-8ac1-3253e552749b-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dsczj\" (UID: \"307fb8b3-6ab3-44b4-8ac1-3253e552749b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.354121 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.354085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/307fb8b3-6ab3-44b4-8ac1-3253e552749b-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dsczj\" (UID: \"307fb8b3-6ab3-44b4-8ac1-3253e552749b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.354301 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.354168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqv5t\" (UniqueName: \"kubernetes.io/projected/307fb8b3-6ab3-44b4-8ac1-3253e552749b-kube-api-access-cqv5t\") pod \"monitoring-plugin-5f68df7f58-dsczj\" (UID: \"307fb8b3-6ab3-44b4-8ac1-3253e552749b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.357193 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.357166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/307fb8b3-6ab3-44b4-8ac1-3253e552749b-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dsczj\" (UID: \"307fb8b3-6ab3-44b4-8ac1-3253e552749b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.362281 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.362248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqv5t\" (UniqueName: \"kubernetes.io/projected/307fb8b3-6ab3-44b4-8ac1-3253e552749b-kube-api-access-cqv5t\") pod \"monitoring-plugin-5f68df7f58-dsczj\" (UID: \"307fb8b3-6ab3-44b4-8ac1-3253e552749b\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.502889 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.502850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:10.517175 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.517145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84494d644c-h2fpg"]
Jul 14 01:01:10.519348 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.519328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.530613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.530593    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84494d644c-h2fpg"]
Jul 14 01:01:10.657135 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-service-ca\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.657326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-serving-cert\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.657326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj8qk\" (UniqueName: \"kubernetes.io/projected/110472b7-680d-4f49-93e4-204c7471eda8-kube-api-access-mj8qk\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.657326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657304    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-console-config\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.657483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657339    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-trusted-ca-bundle\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.657483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657391    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-oauth-config\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.657483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.657446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-oauth-serving-cert\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758434 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-oauth-serving-cert\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758434 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-service-ca\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758951 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-serving-cert\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758951 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj8qk\" (UniqueName: \"kubernetes.io/projected/110472b7-680d-4f49-93e4-204c7471eda8-kube-api-access-mj8qk\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758951 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-console-config\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758951 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-trusted-ca-bundle\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.758951 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.758613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-oauth-config\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.759234 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.759169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-oauth-serving-cert\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.759363 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.759334    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-console-config\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.759508 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.759485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-service-ca\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.760073 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.760030    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-trusted-ca-bundle\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.761353 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.761333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-serving-cert\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.761835 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.761816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-oauth-config\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.767276 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.767252    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj8qk\" (UniqueName: \"kubernetes.io/projected/110472b7-680d-4f49-93e4-204c7471eda8-kube-api-access-mj8qk\") pod \"console-84494d644c-h2fpg\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") " pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:10.830126 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:10.830085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:11.593650 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.593615    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 01:01:11.598130 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.598103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.601646 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.601620    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 01:01:11.601646 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.601636    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4nnj7\""
Jul 14 01:01:11.601877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.601863    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 01:01:11.601932 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.601888    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-eelvb20b7st6c\""
Jul 14 01:01:11.602712 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.602032    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 01:01:11.602712 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.602382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 01:01:11.602712 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.602661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 01:01:11.604089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604066    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 01:01:11.604273 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604253    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 01:01:11.604341 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 01:01:11.605183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604431    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 01:01:11.605183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604551    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 01:01:11.605183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 01:01:11.605183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604551    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 01:01:11.605183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.604792    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 01:01:11.605183 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.605076    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 01:01:11.611084 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.609725    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667711    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-config\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667808    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tljg\" (UniqueName: \"kubernetes.io/projected/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-kube-api-access-5tljg\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667921    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.667978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.668001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669871 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.668033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669871 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.668053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669871 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.668075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.669871 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.668122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.768856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.768856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tljg\" (UniqueName: \"kubernetes.io/projected/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-kube-api-access-5tljg\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.768856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768909    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.768985    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769201    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769253    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769295    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-config\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.769387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.773266 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.773266 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.769971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.773266 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.770215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.773266 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.771121    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.773266 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.771151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.773533 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.773506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.774703 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.774210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.776293 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.776264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.776869 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.776843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-config-out\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.777631 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.777584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.777756 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.777640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.779507 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.778666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.779507 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.779066    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.779507 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.779415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.779507 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.779464    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tljg\" (UniqueName: \"kubernetes.io/projected/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-kube-api-access-5tljg\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.780191 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.779838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-config\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.780191 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.779994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-web-config\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.780191 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.780160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.780328 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.780272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.781286 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.781263    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.915441 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.915398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:11.984495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:11.984251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84c4456fc9-rh78x"]
Jul 14 01:01:12.091351 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:12.091316    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011dd403_949d_4bea_8f16_c3d6e45ef275.slice/crio-af2904bda3d1c6982b9f7c63beb6958c83b7895f8ec4b33ab982e905ba1f1ece WatchSource:0}: Error finding container af2904bda3d1c6982b9f7c63beb6958c83b7895f8ec4b33ab982e905ba1f1ece: Status 404 returned error can't find the container with id af2904bda3d1c6982b9f7c63beb6958c83b7895f8ec4b33ab982e905ba1f1ece
Jul 14 01:01:12.313971 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:12.313041    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84494d644c-h2fpg"]
Jul 14 01:01:12.341339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:12.341252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-645c9f5b98-p4wr9"]
Jul 14 01:01:12.349776 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:12.349743    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"]
Jul 14 01:01:12.402185 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:12.399771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 01:01:12.407862 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:01:12.407141    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb93c8a3_abb6_49aa_9f1c_0c35e9bdd82e.slice/crio-dac5f7675a005fe2626bf438e9a75b3092d53297fb204daa4bba8a8607ea70ac WatchSource:0}: Error finding container dac5f7675a005fe2626bf438e9a75b3092d53297fb204daa4bba8a8607ea70ac: Status 404 returned error can't find the container with id dac5f7675a005fe2626bf438e9a75b3092d53297fb204daa4bba8a8607ea70ac
Jul 14 01:01:13.020744 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.020705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp" event={"ID":"44919429-543c-4426-9b31-bff4ed86fd96","Type":"ContainerStarted","Data":"c38aec943952dc9c7bdecf6fdc4c12772cd4dcb28cea2fc4b53d561ed8c75c66"}
Jul 14 01:01:13.022728 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.022680    2578 generic.go:356] "Generic (PLEG): container finished" podID="a5909f41-52a5-4a5b-8220-f00bd9c71fe5" containerID="d56f09887c4217fbdab384880b6b9d1544444e3219cc8c98e00f85798c8790ce" exitCode=0
Jul 14 01:01:13.022859 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.022784    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerDied","Data":"d56f09887c4217fbdab384880b6b9d1544444e3219cc8c98e00f85798c8790ce"}
Jul 14 01:01:13.024753 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.024731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9" event={"ID":"b4679efc-70e8-4162-a5b2-bf783070fbed","Type":"ContainerStarted","Data":"81bc85cf6a0a45f72b4d289af4eab72ac57f88fa39610f5fc31f2757d1af7d86"}
Jul 14 01:01:13.026702 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.026663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84494d644c-h2fpg" event={"ID":"110472b7-680d-4f49-93e4-204c7471eda8","Type":"ContainerStarted","Data":"10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a"}
Jul 14 01:01:13.026803 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.026717    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84494d644c-h2fpg" event={"ID":"110472b7-680d-4f49-93e4-204c7471eda8","Type":"ContainerStarted","Data":"0976ad87f4152bb1b64fab05b1e6bfa57980875ffdccd3b3a2aadd886fa0243a"}
Jul 14 01:01:13.028582 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.028558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bmkmz" event={"ID":"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b","Type":"ContainerStarted","Data":"8e85ae7be2b93356503f4280aa12618c957bdfe1219d8d202374c9f5fa2e4ba1"}
Jul 14 01:01:13.028680 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.028616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bmkmz" event={"ID":"ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b","Type":"ContainerStarted","Data":"61bab1aebec388cd7ca50456fe350c6781bc7c3d4bb2a37a727df29984cd8db6"}
Jul 14 01:01:13.031595 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.031006    2578 generic.go:356] "Generic (PLEG): container finished" podID="8c0e0edb-e690-47fe-b35c-c8ef1da44b49" containerID="7e3bf1e69fba77263d06f6914246bba56d098c7befd88f1289269001e5ee9b0a" exitCode=0
Jul 14 01:01:13.031595 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.031541    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqlfq" event={"ID":"8c0e0edb-e690-47fe-b35c-c8ef1da44b49","Type":"ContainerDied","Data":"7e3bf1e69fba77263d06f6914246bba56d098c7befd88f1289269001e5ee9b0a"}
Jul 14 01:01:13.034788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.034765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2" event={"ID":"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7","Type":"ContainerStarted","Data":"d0ce901e21ec3f2b53beff28ce526797dcdcfe8b4671a0b7c2c6c70854dee48e"}
Jul 14 01:01:13.034897 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.034814    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2" event={"ID":"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7","Type":"ContainerStarted","Data":"6a4f53b14ab870be17cfab33486f6758753103ca0f4d5f3e919a5d3525b3a52d"}
Jul 14 01:01:13.034897 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.034843    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2" event={"ID":"c741dbc7-4e9c-4677-8f5c-ba80b9804ed7","Type":"ContainerStarted","Data":"ca572fe44baeecebd00eff27740ec2ff9bb379ec957c8e9c461624ea4d946302"}
Jul 14 01:01:13.037988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.036179    2578 generic.go:356] "Generic (PLEG): container finished" podID="cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e" containerID="caa26ad98d7751c7f27fe99c5d61591d182848e76adef48994eb33329d29cad0" exitCode=0
Jul 14 01:01:13.037988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.036451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerDied","Data":"caa26ad98d7751c7f27fe99c5d61591d182848e76adef48994eb33329d29cad0"}
Jul 14 01:01:13.037988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.036511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"dac5f7675a005fe2626bf438e9a75b3092d53297fb204daa4bba8a8607ea70ac"}
Jul 14 01:01:13.038214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.038168    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-pn9vp" podStartSLOduration=2.226459227 podStartE2EDuration="8.038150421s" totalImagesPullingTime="5.811691194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:13.035899958 +0000 UTC m=+74.923164596" watchObservedRunningTime="2026-07-14 01:01:13.038150421 +0000 UTC m=+74.925415051"
Jul 14 01:01:13.038476 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.038422    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kt79b" event={"ID":"447074a6-a97f-4e95-a5d4-3768f6929ba0","Type":"ContainerStarted","Data":"074c57b4b3d5115a4f69588c0706a7f28ada6dd068cbeaf5fc5df75b817f7b4f"}
Jul 14 01:01:13.039999 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.039975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-94xh7" event={"ID":"25529fc8-8cc7-484f-889b-f9b413208193","Type":"ContainerStarted","Data":"a5160850e6c6e686367ba7fed0570e75588367aa7b55c350b290904bf70e1516"}
Jul 14 01:01:13.041226 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.041207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj" event={"ID":"307fb8b3-6ab3-44b4-8ac1-3253e552749b","Type":"ContainerStarted","Data":"727e73131d0d9b819457297c38c1dfba62d9d786a121060d61ba0cfafed98bda"}
Jul 14 01:01:13.041439 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.041423    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:01:13.042656 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.042637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"af2904bda3d1c6982b9f7c63beb6958c83b7895f8ec4b33ab982e905ba1f1ece"}
Jul 14 01:01:13.051637 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.051579    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-4kjk2" podStartSLOduration=2.751345187 podStartE2EDuration="8.051525803s" totalImagesPullingTime="5.300180616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:13.05121178 +0000 UTC m=+74.938476420" watchObservedRunningTime="2026-07-14 01:01:13.051525803 +0000 UTC m=+74.938790434"
Jul 14 01:01:13.081558 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.081448    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bmkmz" podStartSLOduration=68.228459209 podStartE2EDuration="1m15.081430274s" totalImagesPullingTime="6.852971065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:13.080139933 +0000 UTC m=+74.967404560" watchObservedRunningTime="2026-07-14 01:01:13.081430274 +0000 UTC m=+74.968694905"
Jul 14 01:01:13.121392 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.120916    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84494d644c-h2fpg" podStartSLOduration=3.120898493 podStartE2EDuration="3.120898493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:13.119716483 +0000 UTC m=+75.006981113" watchObservedRunningTime="2026-07-14 01:01:13.120898493 +0000 UTC m=+75.008163123"
Jul 14 01:01:13.139605 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.139526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-94xh7" podStartSLOduration=68.83104821 podStartE2EDuration="1m15.139507333s" totalImagesPullingTime="6.308459123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:13.136036888 +0000 UTC m=+75.023301518" watchObservedRunningTime="2026-07-14 01:01:13.139507333 +0000 UTC m=+75.026771963"
Jul 14 01:01:13.152485 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:13.152436    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kt79b" podStartSLOduration=68.172677235 podStartE2EDuration="1m15.152420211s" totalImagesPullingTime="6.979742976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:13.150733128 +0000 UTC m=+75.037997754" watchObservedRunningTime="2026-07-14 01:01:13.152420211 +0000 UTC m=+75.039684840"
Jul 14 01:01:14.051718 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:14.050949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqlfq" event={"ID":"8c0e0edb-e690-47fe-b35c-c8ef1da44b49","Type":"ContainerStarted","Data":"26802784ed1bef41dd80533085bc335ecdd9789e8aa7953c897d945c6a9bcf9d"}
Jul 14 01:01:14.051718 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:14.051043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jqlfq" event={"ID":"8c0e0edb-e690-47fe-b35c-c8ef1da44b49","Type":"ContainerStarted","Data":"673be9ab7aac13434314c189333bf819e459d33d97423026125b588ea797ea01"}
Jul 14 01:01:14.069458 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:14.069406    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jqlfq" podStartSLOduration=3.131400891 podStartE2EDuration="9.069389185s" totalImagesPullingTime="5.783791904s" totalInitContainerRuntime="154.19639ms" isStatefulPod=true podCreationTimestamp="2026-07-14 01:01:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:14.067579569 +0000 UTC m=+75.954844203" watchObservedRunningTime="2026-07-14 01:01:14.069389185 +0000 UTC m=+75.956653813"
Jul 14 01:01:16.061024 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.060992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj" event={"ID":"307fb8b3-6ab3-44b4-8ac1-3253e552749b","Type":"ContainerStarted","Data":"b27a3803435af9eeb7acc60b6b91662bdc9022f68ad2dbbfb027c263d3008a2c"}
Jul 14 01:01:16.061413 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.061189    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:16.063407 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.063378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"7d120ce92daee494e33399bb69f60159b2f387307f7c3a36913fa069c99f3ae5"}
Jul 14 01:01:16.063561 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.063453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"63ee2a034f67562d2234203a3496fcd7a9a16560a2300573291e8d65171eee60"}
Jul 14 01:01:16.063561 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.063491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"fdbd50fe8b55b081176f7b6642cef041ee594cb8ab0ab1eac1466a11f3e87648"}
Jul 14 01:01:16.066735 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.066709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"46d50addcb6c95b594aea14922e480ba0e5df79c4325e063ccb5727d21366f07"}
Jul 14 01:01:16.066846 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.066782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"1fba11ddd8cd96594f2b2d6b2d60bca4f8e5708affadbb2ab0e446ea7eb06a28"}
Jul 14 01:01:16.066846 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.066826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"c4f8b22dfa4c24a56c7784c83f330f70ab66e915672decff1d2d5afad313c5d4"}
Jul 14 01:01:16.066983 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.066911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"90f0cd878c7bae40128669a2ff2ba83fc2b8fce25bed6d1b4bf171d6836dbf52"}
Jul 14 01:01:16.066983 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.066943    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"64fb5dd5d384fd0e773dff6a3f0ef6a07edd9554489557e7582f8272d510b4d6"}
Jul 14 01:01:16.067480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.067461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj"
Jul 14 01:01:16.068453 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.068432    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9" event={"ID":"b4679efc-70e8-4162-a5b2-bf783070fbed","Type":"ContainerStarted","Data":"36b5db83a9bc977b52645e26884f7f0208ad5e90556e478c6caaacc74d4712de"}
Jul 14 01:01:16.076808 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.076766    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dsczj" podStartSLOduration=2.891332493 podStartE2EDuration="6.076753037s" totalImagesPullingTime="3.185420544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:16.075093121 +0000 UTC m=+77.962357751" watchObservedRunningTime="2026-07-14 01:01:16.076753037 +0000 UTC m=+77.964017664"
Jul 14 01:01:16.104103 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:16.104059    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9" podStartSLOduration=3.919872539 podStartE2EDuration="7.10404796s" totalImagesPullingTime="3.184175421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:16.089985105 +0000 UTC m=+77.977249734" watchObservedRunningTime="2026-07-14 01:01:16.10404796 +0000 UTC m=+77.991312588"
Jul 14 01:01:17.075726 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:17.075678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"2984238cfc6944694eddb1017954fd83e05e47aaa3fe21c0f957bc03287ebf0f"}
Jul 14 01:01:17.076085 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:17.075782    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"c35ba2243d9b418cc2d838a867e948e7a32321437112b9fb5874c22f9072a884"}
Jul 14 01:01:18.083734 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:18.083674    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" event={"ID":"011dd403-949d-4bea-8f16-c3d6e45ef275","Type":"ContainerStarted","Data":"4f07d99dfe1bd3a06f730566132c18a94e26a2321bc02f903bab9e130d50b373"}
Jul 14 01:01:18.105604 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:18.105559    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x" podStartSLOduration=5.627762354 podStartE2EDuration="10.105544329s" totalImagesPullingTime="4.477781975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:18.103242975 +0000 UTC m=+79.990507605" watchObservedRunningTime="2026-07-14 01:01:18.105544329 +0000 UTC m=+79.992808960"
Jul 14 01:01:19.091908 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.091860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a5909f41-52a5-4a5b-8220-f00bd9c71fe5","Type":"ContainerStarted","Data":"595eec6947213052227018913c933c5ef67b3841c128be337e43cb8e35cb7f12"}
Jul 14 01:01:19.095214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"8253c4811d9735257bfd03f88ca4404897918ae2ebd65f7355ac2bb8cdf60c27"}
Jul 14 01:01:19.095312 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"dcfdd1be8a98f15459b1702d523857cf3b7f6344f5c3dd9068cbb8d819f317a2"}
Jul 14 01:01:19.095312 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"1b43f6e8c441b5077e838dfcc6531ad0b822e1bbc0d55773df087e007d42c66c"}
Jul 14 01:01:19.095312 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"6f90d137660bb45c75ae515eae37f2ea2aa7e0abe045361ba4889af6794d470a"}
Jul 14 01:01:19.095429 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"f58a48643711d7c59ccd47274be1565c62dffc9a8f0ec2bc75972c697d738259"}
Jul 14 01:01:19.095429 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e","Type":"ContainerStarted","Data":"64fc6ce90095502f9c70e7966e17509659ebb7dfd104d7dd0220359affa52e2c"}
Jul 14 01:01:19.095429 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.095408    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:19.122027 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.121930    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7797553170000002 podStartE2EDuration="13.121915303s" totalImagesPullingTime="10.316768694s" totalInitContainerRuntime="25.391292ms" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:19.120542734 +0000 UTC m=+81.007807355" watchObservedRunningTime="2026-07-14 01:01:19.121915303 +0000 UTC m=+81.009179931"
Jul 14 01:01:19.150829 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:19.150777    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.515526666 podStartE2EDuration="8.150762965s" totalImagesPullingTime="5.601732901s" totalInitContainerRuntime="33.503398ms" isStatefulPod=false podCreationTimestamp="2026-07-14 01:01:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:01:19.148342667 +0000 UTC m=+81.035607298" watchObservedRunningTime="2026-07-14 01:01:19.150762965 +0000 UTC m=+81.038027593"
Jul 14 01:01:20.105059 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:20.105035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84c4456fc9-rh78x"
Jul 14 01:01:20.831084 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:20.831035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:20.831347 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:20.831121    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:20.837425 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:20.837400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:21.112555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:21.112460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:01:21.158190 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:21.158152    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c54499bcb-k4sws"]
Jul 14 01:01:21.915885 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:21.915850    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:01:30.023218 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:30.023181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:30.023796 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:30.023282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:44.054646 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:44.054608    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-94xh7"
Jul 14 01:01:46.181976 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:46.181938    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c54499bcb-k4sws" podUID="93f20eb6-86b6-41c1-b171-b3e64238922f" containerName="console" containerID="cri-o://227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c" gracePeriod=15
Jul 14 01:01:46.974501 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:46.974471    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:01:47.119754 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-service-ca\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.119754 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119732    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc6tb\" (UniqueName: \"kubernetes.io/projected/93f20eb6-86b6-41c1-b171-b3e64238922f-kube-api-access-hc6tb\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.119988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-oauth-serving-cert\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.119988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-oauth-config\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.119988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119900    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-console-config\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.119988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119941    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-trusted-ca-bundle\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.119988 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.119975    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-serving-cert\") pod \"93f20eb6-86b6-41c1-b171-b3e64238922f\" (UID: \"93f20eb6-86b6-41c1-b171-b3e64238922f\") "
Jul 14 01:01:47.120704 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.120649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:01:47.120885 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.120856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-service-ca" (OuterVolumeSpecName: "service-ca") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:01:47.121321 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.121294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:01:47.121629 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.121526    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-console-config" (OuterVolumeSpecName: "console-config") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:01:47.129819 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.129763    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:01:47.130127 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.130015    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:01:47.130346 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.130303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f20eb6-86b6-41c1-b171-b3e64238922f-kube-api-access-hc6tb" (OuterVolumeSpecName: "kube-api-access-hc6tb") pod "93f20eb6-86b6-41c1-b171-b3e64238922f" (UID: "93f20eb6-86b6-41c1-b171-b3e64238922f"). InnerVolumeSpecName "kube-api-access-hc6tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:01:47.220669 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220634    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-console-config\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.220669 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220664    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-trusted-ca-bundle\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.220669 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220675    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-serving-cert\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.221234 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220700    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.221234 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220710    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc6tb\" (UniqueName: \"kubernetes.io/projected/93f20eb6-86b6-41c1-b171-b3e64238922f-kube-api-access-hc6tb\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.221234 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220720    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93f20eb6-86b6-41c1-b171-b3e64238922f-oauth-serving-cert\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.221234 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.220728    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93f20eb6-86b6-41c1-b171-b3e64238922f-console-oauth-config\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:01:47.242974 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.242931    2578 generic.go:356] "Generic (PLEG): container finished" podID="93f20eb6-86b6-41c1-b171-b3e64238922f" containerID="227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c" exitCode=0
Jul 14 01:01:47.243149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.243005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c54499bcb-k4sws" event={"ID":"93f20eb6-86b6-41c1-b171-b3e64238922f","Type":"ContainerDied","Data":"227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c"}
Jul 14 01:01:47.243149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.243055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c54499bcb-k4sws" event={"ID":"93f20eb6-86b6-41c1-b171-b3e64238922f","Type":"ContainerDied","Data":"40b6de7b0837ffb8998b4c80507f3eb5c4ea6c007d1e8e11d3a7389fc93e5750"}
Jul 14 01:01:47.243149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.243084    2578 scope.go:117] "RemoveContainer" containerID="227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c"
Jul 14 01:01:47.243329 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.243259    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c54499bcb-k4sws"
Jul 14 01:01:47.268218 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.268197    2578 scope.go:117] "RemoveContainer" containerID="227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c"
Jul 14 01:01:47.268581 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:01:47.268560    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c\": container with ID starting with 227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c not found: ID does not exist" containerID="227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c"
Jul 14 01:01:47.268658 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.268590    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c"} err="failed to get container status \"227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c\": rpc error: code = NotFound desc = could not find container \"227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c\": container with ID starting with 227eb4dfc571c633af83bcbf43ca7279e8d4de0add1bcd594cd5edabc6cb312c not found: ID does not exist"
Jul 14 01:01:47.271818 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.271791    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c54499bcb-k4sws"]
Jul 14 01:01:47.275597 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:47.275575    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c54499bcb-k4sws"]
Jul 14 01:01:48.690066 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:48.690030    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f20eb6-86b6-41c1-b171-b3e64238922f" path="/var/lib/kubelet/pods/93f20eb6-86b6-41c1-b171-b3e64238922f/volumes"
Jul 14 01:01:50.028720 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:50.028672    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:01:50.032666 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:01:50.032644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-645c9f5b98-p4wr9"
Jul 14 01:02:11.915608 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:02:11.915570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:02:11.937279 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:02:11.937254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:02:12.359383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:02:12.359309    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 01:02:51.608171 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:02:51.608080    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84494d644c-h2fpg"]
Jul 14 01:03:16.629477 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.629411    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84494d644c-h2fpg" podUID="110472b7-680d-4f49-93e4-204c7471eda8" containerName="console" containerID="cri-o://10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a" gracePeriod=15
Jul 14 01:03:16.880805 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.880753    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:03:16.994778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.994751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-console-config\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.994778 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.994783    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-trusted-ca-bundle\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.995014 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.994815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-oauth-config\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.995014 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.994859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mj8qk\" (UniqueName: \"kubernetes.io/projected/110472b7-680d-4f49-93e4-204c7471eda8-kube-api-access-mj8qk\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.995014 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.994915    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-service-ca\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.995174 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-oauth-serving-cert\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.995174 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995115    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-serving-cert\") pod \"110472b7-680d-4f49-93e4-204c7471eda8\" (UID: \"110472b7-680d-4f49-93e4-204c7471eda8\") "
Jul 14 01:03:16.995361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-console-config" (OuterVolumeSpecName: "console-config") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:03:16.995361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995271    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:03:16.995490 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995358    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-service-ca" (OuterVolumeSpecName: "service-ca") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:03:16.995490 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:03:16.995490 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995465    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-console-config\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:16.995490 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995484    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-trusted-ca-bundle\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:16.995715 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.995500    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:16.997229 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.997203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:03:16.997363 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.997345    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110472b7-680d-4f49-93e4-204c7471eda8-kube-api-access-mj8qk" (OuterVolumeSpecName: "kube-api-access-mj8qk") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "kube-api-access-mj8qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:03:16.997433 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:16.997402    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "110472b7-680d-4f49-93e4-204c7471eda8" (UID: "110472b7-680d-4f49-93e4-204c7471eda8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:03:17.096182 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.096150    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-oauth-config\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:17.096182 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.096174    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj8qk\" (UniqueName: \"kubernetes.io/projected/110472b7-680d-4f49-93e4-204c7471eda8-kube-api-access-mj8qk\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:17.096182 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.096184    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/110472b7-680d-4f49-93e4-204c7471eda8-oauth-serving-cert\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:17.096182 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.096192    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/110472b7-680d-4f49-93e4-204c7471eda8-console-serving-cert\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:03:17.586862 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.586825    2578 generic.go:356] "Generic (PLEG): container finished" podID="110472b7-680d-4f49-93e4-204c7471eda8" containerID="10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a" exitCode=0
Jul 14 01:03:17.587057 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.586877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84494d644c-h2fpg"
Jul 14 01:03:17.587057 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.586914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84494d644c-h2fpg" event={"ID":"110472b7-680d-4f49-93e4-204c7471eda8","Type":"ContainerDied","Data":"10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a"}
Jul 14 01:03:17.587057 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.586974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84494d644c-h2fpg" event={"ID":"110472b7-680d-4f49-93e4-204c7471eda8","Type":"ContainerDied","Data":"0976ad87f4152bb1b64fab05b1e6bfa57980875ffdccd3b3a2aadd886fa0243a"}
Jul 14 01:03:17.587057 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.586996    2578 scope.go:117] "RemoveContainer" containerID="10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a"
Jul 14 01:03:17.597525 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.597508    2578 scope.go:117] "RemoveContainer" containerID="10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a"
Jul 14 01:03:17.597800 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:03:17.597782    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a\": container with ID starting with 10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a not found: ID does not exist" containerID="10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a"
Jul 14 01:03:17.597866 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.597810    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a"} err="failed to get container status \"10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a\": rpc error: code = NotFound desc = could not find container \"10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a\": container with ID starting with 10d619d8ad9e5f16b4062e5d6d93d833eddaabca2bd981657c8856ce6cb5e11a not found: ID does not exist"
Jul 14 01:03:17.608857 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.608833    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84494d644c-h2fpg"]
Jul 14 01:03:17.610090 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:17.610072    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84494d644c-h2fpg"]
Jul 14 01:03:18.689890 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:18.689859    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110472b7-680d-4f49-93e4-204c7471eda8" path="/var/lib/kubelet/pods/110472b7-680d-4f49-93e4-204c7471eda8/volumes"
Jul 14 01:03:57.468564 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468531    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hfmh7"]
Jul 14 01:03:57.469046 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468881    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f20eb6-86b6-41c1-b171-b3e64238922f" containerName="console"
Jul 14 01:03:57.469046 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468894    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f20eb6-86b6-41c1-b171-b3e64238922f" containerName="console"
Jul 14 01:03:57.469046 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468906    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110472b7-680d-4f49-93e4-204c7471eda8" containerName="console"
Jul 14 01:03:57.469046 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468911    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="110472b7-680d-4f49-93e4-204c7471eda8" containerName="console"
Jul 14 01:03:57.469046 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468970    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="110472b7-680d-4f49-93e4-204c7471eda8" containerName="console"
Jul 14 01:03:57.469046 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.468980    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f20eb6-86b6-41c1-b171-b3e64238922f" containerName="console"
Jul 14 01:03:57.470815 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.470795    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.472894 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.472874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 14 01:03:57.472990 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.472878    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8dgqw\""
Jul 14 01:03:57.473750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.473732    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 14 01:03:57.481833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.481809    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hfmh7"]
Jul 14 01:03:57.518745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.518720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrkq\" (UniqueName: \"kubernetes.io/projected/085d258e-d341-4a93-86df-52bd34371964-kube-api-access-lrrkq\") pod \"cert-manager-webhook-6486ffd997-hfmh7\" (UID: \"085d258e-d341-4a93-86df-52bd34371964\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.518857 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.518752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/085d258e-d341-4a93-86df-52bd34371964-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hfmh7\" (UID: \"085d258e-d341-4a93-86df-52bd34371964\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.619602 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.619580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrkq\" (UniqueName: \"kubernetes.io/projected/085d258e-d341-4a93-86df-52bd34371964-kube-api-access-lrrkq\") pod \"cert-manager-webhook-6486ffd997-hfmh7\" (UID: \"085d258e-d341-4a93-86df-52bd34371964\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.619712 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.619609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/085d258e-d341-4a93-86df-52bd34371964-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hfmh7\" (UID: \"085d258e-d341-4a93-86df-52bd34371964\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.627290 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.627272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/085d258e-d341-4a93-86df-52bd34371964-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-hfmh7\" (UID: \"085d258e-d341-4a93-86df-52bd34371964\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.627779 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.627762    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrkq\" (UniqueName: \"kubernetes.io/projected/085d258e-d341-4a93-86df-52bd34371964-kube-api-access-lrrkq\") pod \"cert-manager-webhook-6486ffd997-hfmh7\" (UID: \"085d258e-d341-4a93-86df-52bd34371964\") " pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.793214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.793115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:03:57.923096 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:57.922992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-hfmh7"]
Jul 14 01:03:57.925840 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:03:57.925815    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod085d258e_d341_4a93_86df_52bd34371964.slice/crio-218809267e5f5229d7560afd9cb71be98cd05c9428393517fd4aa0fd0198dc72 WatchSource:0}: Error finding container 218809267e5f5229d7560afd9cb71be98cd05c9428393517fd4aa0fd0198dc72: Status 404 returned error can't find the container with id 218809267e5f5229d7560afd9cb71be98cd05c9428393517fd4aa0fd0198dc72
Jul 14 01:03:58.749921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:58.749885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7" event={"ID":"085d258e-d341-4a93-86df-52bd34371964","Type":"ContainerStarted","Data":"218809267e5f5229d7560afd9cb71be98cd05c9428393517fd4aa0fd0198dc72"}
Jul 14 01:03:59.156518 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.156442    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"]
Jul 14 01:03:59.159427 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.159411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.161709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.161670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wpmpj\""
Jul 14 01:03:59.167134 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.167113    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"]
Jul 14 01:03:59.231914 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.231882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fcd9391-c940-4a99-b804-6efce785701e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rjhkc\" (UID: \"2fcd9391-c940-4a99-b804-6efce785701e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.231914 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.231918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klhq4\" (UniqueName: \"kubernetes.io/projected/2fcd9391-c940-4a99-b804-6efce785701e-kube-api-access-klhq4\") pod \"cert-manager-cainjector-559cdd8c87-rjhkc\" (UID: \"2fcd9391-c940-4a99-b804-6efce785701e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.332877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.332830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fcd9391-c940-4a99-b804-6efce785701e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rjhkc\" (UID: \"2fcd9391-c940-4a99-b804-6efce785701e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.332877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.332881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klhq4\" (UniqueName: \"kubernetes.io/projected/2fcd9391-c940-4a99-b804-6efce785701e-kube-api-access-klhq4\") pod \"cert-manager-cainjector-559cdd8c87-rjhkc\" (UID: \"2fcd9391-c940-4a99-b804-6efce785701e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.341568 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.341535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fcd9391-c940-4a99-b804-6efce785701e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-rjhkc\" (UID: \"2fcd9391-c940-4a99-b804-6efce785701e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.341709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.341625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klhq4\" (UniqueName: \"kubernetes.io/projected/2fcd9391-c940-4a99-b804-6efce785701e-kube-api-access-klhq4\") pod \"cert-manager-cainjector-559cdd8c87-rjhkc\" (UID: \"2fcd9391-c940-4a99-b804-6efce785701e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.469271 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.469237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"
Jul 14 01:03:59.591559 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.591528    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc"]
Jul 14 01:03:59.594398 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:03:59.594368    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fcd9391_c940_4a99_b804_6efce785701e.slice/crio-ff78d28da1b8933bff8c4dba280eb7908b465e0f7ac25d575b1b69064d6621ac WatchSource:0}: Error finding container ff78d28da1b8933bff8c4dba280eb7908b465e0f7ac25d575b1b69064d6621ac: Status 404 returned error can't find the container with id ff78d28da1b8933bff8c4dba280eb7908b465e0f7ac25d575b1b69064d6621ac
Jul 14 01:03:59.754835 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:03:59.754754    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc" event={"ID":"2fcd9391-c940-4a99-b804-6efce785701e","Type":"ContainerStarted","Data":"ff78d28da1b8933bff8c4dba280eb7908b465e0f7ac25d575b1b69064d6621ac"}
Jul 14 01:04:04.784403 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:04.784363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7" event={"ID":"085d258e-d341-4a93-86df-52bd34371964","Type":"ContainerStarted","Data":"402527d5d82ac1c57ee79927ef4e2c57214aae4b651076b80c5e4cb2d4f2be7c"}
Jul 14 01:04:04.784875 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:04.784419    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:04:04.785790 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:04.785767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc" event={"ID":"2fcd9391-c940-4a99-b804-6efce785701e","Type":"ContainerStarted","Data":"b75fe00e030c539e6983037a5c68a8c62899a26e1bd8dd9e1cf36eadfff06b6c"}
Jul 14 01:04:04.801070 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:04.801023    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7" podStartSLOduration=1.827539935 podStartE2EDuration="7.801007235s" totalImagesPullingTime="5.9734673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:03:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:04.799633993 +0000 UTC m=+246.686898620" watchObservedRunningTime="2026-07-14 01:04:04.801007235 +0000 UTC m=+246.688271863"
Jul 14 01:04:04.816568 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:04.816529    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-rjhkc" podStartSLOduration=1.502402335 podStartE2EDuration="5.816516975s" totalImagesPullingTime="4.31411464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:04:04.815431919 +0000 UTC m=+246.702696547" watchObservedRunningTime="2026-07-14 01:04:04.816516975 +0000 UTC m=+246.703781604"
Jul 14 01:04:10.796997 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:10.796964    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-hfmh7"
Jul 14 01:04:58.507242 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:58.507211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:04:58.508582 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:58.508552    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:04:58.512423 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:04:58.512401    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 01:06:18.622618 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.622584    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"]
Jul 14 01:06:18.625723 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.625706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.628193 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.628167    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 01:06:18.628348 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.628178    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 01:06:18.629032 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.628985    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 01:06:18.629179 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.629058    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 01:06:18.629179 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.629085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gq2vk\""
Jul 14 01:06:18.629179 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.629152    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 01:06:18.633600 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.633580    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"]
Jul 14 01:06:18.786985 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.786949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea48cf76-3f3b-4df9-8897-c568f12a8012-cert\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.786985 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.786986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea48cf76-3f3b-4df9-8897-c568f12a8012-metrics-cert\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.787187 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.787012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea48cf76-3f3b-4df9-8897-c568f12a8012-manager-config\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.787187 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.787092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fz8f\" (UniqueName: \"kubernetes.io/projected/ea48cf76-3f3b-4df9-8897-c568f12a8012-kube-api-access-6fz8f\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.888472 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.888374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea48cf76-3f3b-4df9-8897-c568f12a8012-cert\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.888472 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.888410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea48cf76-3f3b-4df9-8897-c568f12a8012-metrics-cert\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.888472 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.888436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea48cf76-3f3b-4df9-8897-c568f12a8012-manager-config\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.888788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.888508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fz8f\" (UniqueName: \"kubernetes.io/projected/ea48cf76-3f3b-4df9-8897-c568f12a8012-kube-api-access-6fz8f\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.889456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.889255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ea48cf76-3f3b-4df9-8897-c568f12a8012-manager-config\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.891174 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.891144    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea48cf76-3f3b-4df9-8897-c568f12a8012-metrics-cert\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.891289 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.891253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ea48cf76-3f3b-4df9-8897-c568f12a8012-cert\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.896625 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.896605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fz8f\" (UniqueName: \"kubernetes.io/projected/ea48cf76-3f3b-4df9-8897-c568f12a8012-kube-api-access-6fz8f\") pod \"lws-controller-manager-7cb746fd84-4zjm5\" (UID: \"ea48cf76-3f3b-4df9-8897-c568f12a8012\") " pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:18.938058 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:18.938030    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:19.071385 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:19.071361    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"]
Jul 14 01:06:19.073461 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:06:19.073436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea48cf76_3f3b_4df9_8897_c568f12a8012.slice/crio-8a5a87b84331752d896a97fae05245bacb066ed94f7aa34f3c76a65d92af17f5 WatchSource:0}: Error finding container 8a5a87b84331752d896a97fae05245bacb066ed94f7aa34f3c76a65d92af17f5: Status 404 returned error can't find the container with id 8a5a87b84331752d896a97fae05245bacb066ed94f7aa34f3c76a65d92af17f5
Jul 14 01:06:19.075444 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:19.075429    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:06:19.310040 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:19.310006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5" event={"ID":"ea48cf76-3f3b-4df9-8897-c568f12a8012","Type":"ContainerStarted","Data":"8a5a87b84331752d896a97fae05245bacb066ed94f7aa34f3c76a65d92af17f5"}
Jul 14 01:06:24.332543 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:24.332500    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5" event={"ID":"ea48cf76-3f3b-4df9-8897-c568f12a8012","Type":"ContainerStarted","Data":"3965a55a7ce433dbd5efcbf0dc1353582304d8446fc25a755d64943ec8bd7622"}
Jul 14 01:06:24.332948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:24.332563    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:24.347164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:24.347110    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5" podStartSLOduration=1.273912463 podStartE2EDuration="6.347091779s" totalImagesPullingTime="5.073179316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:06:24.346534041 +0000 UTC m=+386.233798669" watchObservedRunningTime="2026-07-14 01:06:24.347091779 +0000 UTC m=+386.234356410"
Jul 14 01:06:28.866295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.866256    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"]
Jul 14 01:06:28.869941 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.869919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:28.872534 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.872506    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 01:06:28.872677 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.872597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 01:06:28.872890 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.872868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 01:06:28.872974 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.872940    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 01:06:28.873036 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.872869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-pscwn\""
Jul 14 01:06:28.884923 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.884901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"]
Jul 14 01:06:28.978353 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.978322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/721178c4-28cb-40a6-b4dc-45407dafbd9b-tmp\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:28.978353 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.978353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpstd\" (UniqueName: \"kubernetes.io/projected/721178c4-28cb-40a6-b4dc-45407dafbd9b-kube-api-access-fpstd\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:28.978531 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.978392    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/721178c4-28cb-40a6-b4dc-45407dafbd9b-manifests\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:28.978531 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.978418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/721178c4-28cb-40a6-b4dc-45407dafbd9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:28.978531 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:28.978449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/721178c4-28cb-40a6-b4dc-45407dafbd9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.079250 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.079215    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/721178c4-28cb-40a6-b4dc-45407dafbd9b-tmp\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.079250 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.079250    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpstd\" (UniqueName: \"kubernetes.io/projected/721178c4-28cb-40a6-b4dc-45407dafbd9b-kube-api-access-fpstd\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.079496 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.079286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/721178c4-28cb-40a6-b4dc-45407dafbd9b-manifests\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.079496 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.079413    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/721178c4-28cb-40a6-b4dc-45407dafbd9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.079496 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.079477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/721178c4-28cb-40a6-b4dc-45407dafbd9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.079661 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.079641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/721178c4-28cb-40a6-b4dc-45407dafbd9b-manifests\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.081735 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.081715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/721178c4-28cb-40a6-b4dc-45407dafbd9b-tmp\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.081866 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.081852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/721178c4-28cb-40a6-b4dc-45407dafbd9b-webhook-cert\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.081907 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.081887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/721178c4-28cb-40a6-b4dc-45407dafbd9b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.087042 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.087020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpstd\" (UniqueName: \"kubernetes.io/projected/721178c4-28cb-40a6-b4dc-45407dafbd9b-kube-api-access-fpstd\") pod \"opendatahub-operator-controller-manager-554b6949c8-6dqbl\" (UID: \"721178c4-28cb-40a6-b4dc-45407dafbd9b\") " pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.182204 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.182172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:29.322025 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.321993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"]
Jul 14 01:06:29.326037 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:06:29.325996    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721178c4_28cb_40a6_b4dc_45407dafbd9b.slice/crio-465f596ead36cd04fdfb58c54fac0f13c6cecc41f8b4f0bc9d0fe7e5f6334b7c WatchSource:0}: Error finding container 465f596ead36cd04fdfb58c54fac0f13c6cecc41f8b4f0bc9d0fe7e5f6334b7c: Status 404 returned error can't find the container with id 465f596ead36cd04fdfb58c54fac0f13c6cecc41f8b4f0bc9d0fe7e5f6334b7c
Jul 14 01:06:29.354237 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:29.354201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl" event={"ID":"721178c4-28cb-40a6-b4dc-45407dafbd9b","Type":"ContainerStarted","Data":"465f596ead36cd04fdfb58c54fac0f13c6cecc41f8b4f0bc9d0fe7e5f6334b7c"}
Jul 14 01:06:33.378945 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:33.378914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl" event={"ID":"721178c4-28cb-40a6-b4dc-45407dafbd9b","Type":"ContainerStarted","Data":"0ad8959a422458fbf153581d3da7d3e3b250a2fdd4bad09c7dfaf7defd827e1a"}
Jul 14 01:06:34.385177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:34.385142    2578 generic.go:356] "Generic (PLEG): container finished" podID="721178c4-28cb-40a6-b4dc-45407dafbd9b" containerID="0ad8959a422458fbf153581d3da7d3e3b250a2fdd4bad09c7dfaf7defd827e1a" exitCode=0
Jul 14 01:06:34.385589 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:34.385241    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl" event={"ID":"721178c4-28cb-40a6-b4dc-45407dafbd9b","Type":"ContainerDied","Data":"0ad8959a422458fbf153581d3da7d3e3b250a2fdd4bad09c7dfaf7defd827e1a"}
Jul 14 01:06:35.339017 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:35.338989    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cb746fd84-4zjm5"
Jul 14 01:06:35.392359 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:35.392321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl" event={"ID":"721178c4-28cb-40a6-b4dc-45407dafbd9b","Type":"ContainerStarted","Data":"071a6f4f9bc8c148a42351f74ba86a20386f50de724ce672e46292b411b6ab3e"}
Jul 14 01:06:35.392780 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:35.392475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:35.413650 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:35.413602    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl" podStartSLOduration=2.835655483 podStartE2EDuration="7.413587355s" totalImagesPullingTime="3.501005912s" totalInitContainerRuntime="1.07692596s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:06:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:06:35.412269123 +0000 UTC m=+397.299533750" watchObservedRunningTime="2026-07-14 01:06:35.413587355 +0000 UTC m=+397.300851982"
Jul 14 01:06:46.398415 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:46.398387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-554b6949c8-6dqbl"
Jul 14 01:06:54.161092 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.161054    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"]
Jul 14 01:06:54.164823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.164796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.167263 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.167232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 01:06:54.167383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.167271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 01:06:54.167383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.167351    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 01:06:54.167517 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.167482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 01:06:54.168189 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.168168    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x48rj\""
Jul 14 01:06:54.176616 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.176596    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"]
Jul 14 01:06:54.292750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.292680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl6c7\" (UniqueName: \"kubernetes.io/projected/fb370852-4001-4d24-aebe-063138ee8c2c-kube-api-access-kl6c7\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.292936 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.292814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb370852-4001-4d24-aebe-063138ee8c2c-tls-certs\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.292936 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.292843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb370852-4001-4d24-aebe-063138ee8c2c-tmp\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.393637 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.393598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl6c7\" (UniqueName: \"kubernetes.io/projected/fb370852-4001-4d24-aebe-063138ee8c2c-kube-api-access-kl6c7\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.393901 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.393718    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb370852-4001-4d24-aebe-063138ee8c2c-tls-certs\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.393901 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.393752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb370852-4001-4d24-aebe-063138ee8c2c-tmp\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.393901 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:06:54.393869    2578 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 14 01:06:54.394196 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:06:54.393952    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb370852-4001-4d24-aebe-063138ee8c2c-tls-certs podName:fb370852-4001-4d24-aebe-063138ee8c2c nodeName:}" failed. No retries permitted until 2026-07-14 01:06:54.893928452 +0000 UTC m=+416.781193058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fb370852-4001-4d24-aebe-063138ee8c2c-tls-certs") pod "kube-auth-proxy-54d956c5b-pdkbg" (UID: "fb370852-4001-4d24-aebe-063138ee8c2c") : secret "kube-auth-proxy-tls" not found
Jul 14 01:06:54.396288 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.396265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb370852-4001-4d24-aebe-063138ee8c2c-tmp\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.405399 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.405377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl6c7\" (UniqueName: \"kubernetes.io/projected/fb370852-4001-4d24-aebe-063138ee8c2c-kube-api-access-kl6c7\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.898796 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.898750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb370852-4001-4d24-aebe-063138ee8c2c-tls-certs\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:54.901503 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:54.901473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb370852-4001-4d24-aebe-063138ee8c2c-tls-certs\") pod \"kube-auth-proxy-54d956c5b-pdkbg\" (UID: \"fb370852-4001-4d24-aebe-063138ee8c2c\") " pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:55.076567 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:55.076537    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"
Jul 14 01:06:55.225440 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:55.225418    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg"]
Jul 14 01:06:55.227654 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:06:55.227626    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb370852_4001_4d24_aebe_063138ee8c2c.slice/crio-b89d7a9a2b5d4fd046d84f68167720afc958c0fb48e19be26b495e3dcfd24033 WatchSource:0}: Error finding container b89d7a9a2b5d4fd046d84f68167720afc958c0fb48e19be26b495e3dcfd24033: Status 404 returned error can't find the container with id b89d7a9a2b5d4fd046d84f68167720afc958c0fb48e19be26b495e3dcfd24033
Jul 14 01:06:55.486216 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:55.486143    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg" event={"ID":"fb370852-4001-4d24-aebe-063138ee8c2c","Type":"ContainerStarted","Data":"b89d7a9a2b5d4fd046d84f68167720afc958c0fb48e19be26b495e3dcfd24033"}
Jul 14 01:06:58.508739 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:58.508699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg" event={"ID":"fb370852-4001-4d24-aebe-063138ee8c2c","Type":"ContainerStarted","Data":"d54de58483a99be834383b80b2f472eef4568c1fd16dafb045faaaf9ba282cd9"}
Jul 14 01:06:58.526659 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:06:58.526609    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54d956c5b-pdkbg" podStartSLOduration=1.329014812 podStartE2EDuration="4.526594852s" totalImagesPullingTime="3.19758004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:06:58.524750282 +0000 UTC m=+420.412014911" watchObservedRunningTime="2026-07-14 01:06:58.526594852 +0000 UTC m=+420.413859480"
Jul 14 01:08:27.408637 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.408600    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"]
Jul 14 01:08:27.411546 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.411525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:27.413925 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.413899    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 01:08:27.414590 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.414574    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wb28v\""
Jul 14 01:08:27.414702 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.414573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 01:08:27.421355 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.421322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"]
Jul 14 01:08:27.484874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.484846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grkf6\" (UniqueName: \"kubernetes.io/projected/d4adcc74-e74a-4ac0-812e-0bebf57a5022-kube-api-access-grkf6\") pod \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" (UID: \"d4adcc74-e74a-4ac0-812e-0bebf57a5022\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:27.585807 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.585771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grkf6\" (UniqueName: \"kubernetes.io/projected/d4adcc74-e74a-4ac0-812e-0bebf57a5022-kube-api-access-grkf6\") pod \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" (UID: \"d4adcc74-e74a-4ac0-812e-0bebf57a5022\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:27.599161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.599130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grkf6\" (UniqueName: \"kubernetes.io/projected/d4adcc74-e74a-4ac0-812e-0bebf57a5022-kube-api-access-grkf6\") pod \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" (UID: \"d4adcc74-e74a-4ac0-812e-0bebf57a5022\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:27.723112 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.723080    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:27.855910 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.855860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"]
Jul 14 01:08:27.858250 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:08:27.858220    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4adcc74_e74a_4ac0_812e_0bebf57a5022.slice/crio-f53a9cc88c97295eeb608ae77369f3045e1b082c18cdec30a47671596110de68 WatchSource:0}: Error finding container f53a9cc88c97295eeb608ae77369f3045e1b082c18cdec30a47671596110de68: Status 404 returned error can't find the container with id f53a9cc88c97295eeb608ae77369f3045e1b082c18cdec30a47671596110de68
Jul 14 01:08:27.889718 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:27.889675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z" event={"ID":"d4adcc74-e74a-4ac0-812e-0bebf57a5022","Type":"ContainerStarted","Data":"f53a9cc88c97295eeb608ae77369f3045e1b082c18cdec30a47671596110de68"}
Jul 14 01:08:30.905081 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:30.905049    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z" event={"ID":"d4adcc74-e74a-4ac0-812e-0bebf57a5022","Type":"ContainerStarted","Data":"8a5e4e1c369a116a48f9ea0a5d2500c4595aa62ce86cd7c9dc3aa5fa9300c155"}
Jul 14 01:08:30.905470 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:30.905111    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:30.922432 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:30.922385    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z" podStartSLOduration=1.093204773 podStartE2EDuration="3.922371352s" totalImagesPullingTime="2.829166579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:30.919940707 +0000 UTC m=+512.807205356" watchObservedRunningTime="2026-07-14 01:08:30.922371352 +0000 UTC m=+512.809635991"
Jul 14 01:08:41.737322 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.737287    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"]
Jul 14 01:08:41.737935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.737554    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" containerName="manager" containerID="cri-o://8a5e4e1c369a116a48f9ea0a5d2500c4595aa62ce86cd7c9dc3aa5fa9300c155" gracePeriod=2
Jul 14 01:08:41.739972 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.739946    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:41.740475 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:08:41.740439    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:41.740675 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.740646    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"]
Jul 14 01:08:41.742206 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:08:41.742175    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:41.763435 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.763405    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"]
Jul 14 01:08:41.763781 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.763766    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" containerName="manager"
Jul 14 01:08:41.763781 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.763781    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" containerName="manager"
Jul 14 01:08:41.763939 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.763836    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" containerName="manager"
Jul 14 01:08:41.765859 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.765836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:08:41.778095 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.778066    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"]
Jul 14 01:08:41.800862 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:08:41.800829    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:41.816511 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.816487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zmtt\" (UniqueName: \"kubernetes.io/projected/abfa3bfb-585d-4c34-aae2-fe93c372cb69-kube-api-access-4zmtt\") pod \"limitador-operator-controller-manager-859d7b78bd-nbhbd\" (UID: \"abfa3bfb-585d-4c34-aae2-fe93c372cb69\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:08:41.917741 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.917709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zmtt\" (UniqueName: \"kubernetes.io/projected/abfa3bfb-585d-4c34-aae2-fe93c372cb69-kube-api-access-4zmtt\") pod \"limitador-operator-controller-manager-859d7b78bd-nbhbd\" (UID: \"abfa3bfb-585d-4c34-aae2-fe93c372cb69\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:08:41.931301 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.931268    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zmtt\" (UniqueName: \"kubernetes.io/projected/abfa3bfb-585d-4c34-aae2-fe93c372cb69-kube-api-access-4zmtt\") pod \"limitador-operator-controller-manager-859d7b78bd-nbhbd\" (UID: \"abfa3bfb-585d-4c34-aae2-fe93c372cb69\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:08:41.953024 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.952992    2578 generic.go:356] "Generic (PLEG): container finished" podID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" containerID="8a5e4e1c369a116a48f9ea0a5d2500c4595aa62ce86cd7c9dc3aa5fa9300c155" exitCode=0
Jul 14 01:08:41.975075 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:41.975055    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:41.977162 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:08:41.977134    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pjp9z\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:42.119328 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.119246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grkf6\" (UniqueName: \"kubernetes.io/projected/d4adcc74-e74a-4ac0-812e-0bebf57a5022-kube-api-access-grkf6\") pod \"d4adcc74-e74a-4ac0-812e-0bebf57a5022\" (UID: \"d4adcc74-e74a-4ac0-812e-0bebf57a5022\") "
Jul 14 01:08:42.121665 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.121640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4adcc74-e74a-4ac0-812e-0bebf57a5022-kube-api-access-grkf6" (OuterVolumeSpecName: "kube-api-access-grkf6") pod "d4adcc74-e74a-4ac0-812e-0bebf57a5022" (UID: "d4adcc74-e74a-4ac0-812e-0bebf57a5022"). InnerVolumeSpecName "kube-api-access-grkf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:08:42.130790 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.130768    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:08:42.220253 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.220224    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grkf6\" (UniqueName: \"kubernetes.io/projected/d4adcc74-e74a-4ac0-812e-0bebf57a5022-kube-api-access-grkf6\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:08:42.283740 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.283714    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"]
Jul 14 01:08:42.285341 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:08:42.285311    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabfa3bfb_585d_4c34_aae2_fe93c372cb69.slice/crio-ace7a24fcd8485c9b2fca8259d2a8788e7c24b5737ed2251f97120d70582884c WatchSource:0}: Error finding container ace7a24fcd8485c9b2fca8259d2a8788e7c24b5737ed2251f97120d70582884c: Status 404 returned error can't find the container with id ace7a24fcd8485c9b2fca8259d2a8788e7c24b5737ed2251f97120d70582884c
Jul 14 01:08:42.690340 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.690305    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4adcc74-e74a-4ac0-812e-0bebf57a5022" path="/var/lib/kubelet/pods/d4adcc74-e74a-4ac0-812e-0bebf57a5022/volumes"
Jul 14 01:08:42.958884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.958808    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pjp9z"
Jul 14 01:08:42.958884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.958817    2578 scope.go:117] "RemoveContainer" containerID="8a5e4e1c369a116a48f9ea0a5d2500c4595aa62ce86cd7c9dc3aa5fa9300c155"
Jul 14 01:08:42.960737 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.960588    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd" event={"ID":"abfa3bfb-585d-4c34-aae2-fe93c372cb69","Type":"ContainerStarted","Data":"bc1c5a30bc08b1b4cd123813c14e566b177b3b4fb98ebab7a45cdb55ab1064d0"}
Jul 14 01:08:42.960737 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.960648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd" event={"ID":"abfa3bfb-585d-4c34-aae2-fe93c372cb69","Type":"ContainerStarted","Data":"ace7a24fcd8485c9b2fca8259d2a8788e7c24b5737ed2251f97120d70582884c"}
Jul 14 01:08:42.960737 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.960726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:08:42.977436 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:42.977345    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd" podStartSLOduration=1.977329354 podStartE2EDuration="1.977329354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:08:42.976866782 +0000 UTC m=+524.864131411" watchObservedRunningTime="2026-07-14 01:08:42.977329354 +0000 UTC m=+524.864593982"
Jul 14 01:08:53.968588 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:08:53.968509    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nbhbd"
Jul 14 01:09:22.731924 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.731886    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:09:22.735209 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.735180    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:22.737439 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.737416    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 14 01:09:22.737579 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.737419    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rc5dh\""
Jul 14 01:09:22.742219 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.742198    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:09:22.770074 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.770044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:09:22.770402 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.770380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xw7j\" (UniqueName: \"kubernetes.io/projected/080da3cc-39fd-4eed-80e4-7c37429ccbe2-kube-api-access-7xw7j\") pod \"limitador-limitador-54d6d988d5-xpdt7\" (UID: \"080da3cc-39fd-4eed-80e4-7c37429ccbe2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:22.770493 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.770479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/080da3cc-39fd-4eed-80e4-7c37429ccbe2-config-file\") pod \"limitador-limitador-54d6d988d5-xpdt7\" (UID: \"080da3cc-39fd-4eed-80e4-7c37429ccbe2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:22.872110 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.872064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xw7j\" (UniqueName: \"kubernetes.io/projected/080da3cc-39fd-4eed-80e4-7c37429ccbe2-kube-api-access-7xw7j\") pod \"limitador-limitador-54d6d988d5-xpdt7\" (UID: \"080da3cc-39fd-4eed-80e4-7c37429ccbe2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:22.872281 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.872130    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/080da3cc-39fd-4eed-80e4-7c37429ccbe2-config-file\") pod \"limitador-limitador-54d6d988d5-xpdt7\" (UID: \"080da3cc-39fd-4eed-80e4-7c37429ccbe2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:22.872789 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.872770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/080da3cc-39fd-4eed-80e4-7c37429ccbe2-config-file\") pod \"limitador-limitador-54d6d988d5-xpdt7\" (UID: \"080da3cc-39fd-4eed-80e4-7c37429ccbe2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:22.879170 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:22.879147    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xw7j\" (UniqueName: \"kubernetes.io/projected/080da3cc-39fd-4eed-80e4-7c37429ccbe2-kube-api-access-7xw7j\") pod \"limitador-limitador-54d6d988d5-xpdt7\" (UID: \"080da3cc-39fd-4eed-80e4-7c37429ccbe2\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:23.046936 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:23.046857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:23.178292 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:23.178265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:09:23.180801 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:09:23.180773    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080da3cc_39fd_4eed_80e4_7c37429ccbe2.slice/crio-a06327c5c9ae998c4322d33070177a791e5a4adc3491482ce11303dce3b234de WatchSource:0}: Error finding container a06327c5c9ae998c4322d33070177a791e5a4adc3491482ce11303dce3b234de: Status 404 returned error can't find the container with id a06327c5c9ae998c4322d33070177a791e5a4adc3491482ce11303dce3b234de
Jul 14 01:09:24.145857 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:24.145822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7" event={"ID":"080da3cc-39fd-4eed-80e4-7c37429ccbe2","Type":"ContainerStarted","Data":"a06327c5c9ae998c4322d33070177a791e5a4adc3491482ce11303dce3b234de"}
Jul 14 01:09:27.168612 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:27.168577    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7" event={"ID":"080da3cc-39fd-4eed-80e4-7c37429ccbe2","Type":"ContainerStarted","Data":"f44ab393c71e7d176e8e6abe9e2edccaf628c09613114ca4afa65986851db51f"}
Jul 14 01:09:27.168989 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:27.168652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:27.187417 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:27.187370    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7" podStartSLOduration=1.29755665 podStartE2EDuration="5.187355083s" totalImagesPullingTime="3.889798433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:09:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:09:27.185522149 +0000 UTC m=+569.072786778" watchObservedRunningTime="2026-07-14 01:09:27.187355083 +0000 UTC m=+569.074619749"
Jul 14 01:09:38.174967 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:38.174935    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"
Jul 14 01:09:58.538962 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:58.538937    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:09:58.540166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:09:58.540145    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:10:46.380874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:10:46.380794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:11:10.332637 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:11:10.332607    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:11:20.338103 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:11:20.338071    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:11:35.330361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:11:35.330324    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:11:48.830947 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:11:48.830860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:11:59.532764 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:11:59.532732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:12:12.319223 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.319192    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"]
Jul 14 01:12:12.322756 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.322735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.325823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.325801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 01:12:12.325823 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.325815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 01:12:12.326012 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.325815    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-48wgh\""
Jul 14 01:12:12.326012 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.325803    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 01:12:12.333857 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.333833    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"]
Jul 14 01:12:12.419314 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.419285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.419314 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.419316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.419495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.419347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.419495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.419403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.419495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.419433    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d375ca4-2afa-4099-9458-fc4834f38a39-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.419495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.419466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsvc2\" (UniqueName: \"kubernetes.io/projected/8d375ca4-2afa-4099-9458-fc4834f38a39-kube-api-access-hsvc2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520051 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d375ca4-2afa-4099-9458-fc4834f38a39-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520208 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsvc2\" (UniqueName: \"kubernetes.io/projected/8d375ca4-2afa-4099-9458-fc4834f38a39-kube-api-access-hsvc2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520208 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520099    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520208 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520208 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520145    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520208 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520547 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520547 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520542    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.520661 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.520626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.522514 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.522494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d375ca4-2afa-4099-9458-fc4834f38a39-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.522767 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.522750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d375ca4-2afa-4099-9458-fc4834f38a39-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.531463 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.531442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsvc2\" (UniqueName: \"kubernetes.io/projected/8d375ca4-2afa-4099-9458-fc4834f38a39-kube-api-access-hsvc2\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj\" (UID: \"8d375ca4-2afa-4099-9458-fc4834f38a39\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.633810 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.633725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:12.763763 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.763740    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"]
Jul 14 01:12:12.766123 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:12:12.766096    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d375ca4_2afa_4099_9458_fc4834f38a39.slice/crio-941de0033366ded12c083b15ef86584795fd2acecd102ffc5d6a281fd1aaef10 WatchSource:0}: Error finding container 941de0033366ded12c083b15ef86584795fd2acecd102ffc5d6a281fd1aaef10: Status 404 returned error can't find the container with id 941de0033366ded12c083b15ef86584795fd2acecd102ffc5d6a281fd1aaef10
Jul 14 01:12:12.767461 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.767443    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:12:12.905405 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:12.905325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj" event={"ID":"8d375ca4-2afa-4099-9458-fc4834f38a39","Type":"ContainerStarted","Data":"941de0033366ded12c083b15ef86584795fd2acecd102ffc5d6a281fd1aaef10"}
Jul 14 01:12:13.912318 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:13.912286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj" event={"ID":"8d375ca4-2afa-4099-9458-fc4834f38a39","Type":"ContainerStarted","Data":"8e315ec482f621c9a032138cfdfd1a5faeac5671c919a9b28765186bd67d7914"}
Jul 14 01:12:13.928673 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:13.928620    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj" podStartSLOduration=0.894282909 podStartE2EDuration="1.928600013s" totalImagesPullingTime="1.034317104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:12:13.927507749 +0000 UTC m=+735.814772380" watchObservedRunningTime="2026-07-14 01:12:13.928600013 +0000 UTC m=+735.815864642"
Jul 14 01:12:15.157270 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:15.157233    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:12:22.634200 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:22.634163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:22.634627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:22.634216    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:22.646625 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:22.646607    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:12:22.965801 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:12:22.965772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj"
Jul 14 01:13:35.332295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:13:35.332208    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:13:44.332923 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:13:44.332882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:13:56.434786 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:13:56.434753    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:14:05.730278 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:14:05.730244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:14:16.532639 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:14:16.532603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:14:43.226349 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:14:43.226309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:14:55.334983 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:14:55.334903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:14:58.572969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:14:58.572937    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:14:58.575174 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:14:58.575153    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:15:30.435334 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:15:30.435297    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:15:41.330058 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:15:41.330027    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:15:52.726293 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:15:52.726260    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:16:03.232917 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:16:03.232883    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:16:14.439802 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:16:14.439771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:16:23.734569 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:16:23.734494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:16:34.731144 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:16:34.731108    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:16:44.730528 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:16:44.730491    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:17:10.431533 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:17:10.431504    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:17:21.234660 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:17:21.234631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:17:42.926655 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:17:42.926619    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:17:53.029226 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:17:53.029146    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:18:09.273657 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:18:09.273614    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:18:18.835937 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:18:18.835905    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:18:28.324354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:18:28.324317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:18:37.634945 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:18:37.634913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:18:46.579681 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:18:46.579644    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:19:58.608050 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:19:58.608011    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:19:58.614260 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:19:58.614242    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:20:37.732841 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:20:37.732808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:20:47.334536 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:20:47.334498    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:20:58.743032 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:20:58.742951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:21:09.527540 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:21:09.527502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:21:19.431731 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:21:19.431675    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:21:30.729877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:21:30.729840    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:21:39.732996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:21:39.732957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:21:50.335759 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:21:50.335726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:22:01.128653 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:22:01.128618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:22:10.335115 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:22:10.335085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:22:21.429435 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:22:21.429342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:22:42.930443 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:22:42.930405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:22:52.133254 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:22:52.133220    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:23:01.339716 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:23:01.337087    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:23:13.728898 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:23:13.728861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:23:27.931797 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:23:27.931763    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:23:42.327482 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:23:42.327437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:23:51.032504 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:23:51.032408    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:24:06.037280 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:06.037245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:24:15.630617 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:15.630581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:24:29.239499 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:29.239449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:24:38.937640 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:38.937604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:24:48.333389 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:48.333355    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:24:58.639248 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:58.639222    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:24:58.645108 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:24:58.645091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:25:00.935408 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:25:00.935373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:25:11.431741 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:25:11.431706    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:25:19.729193 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:25:19.729117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:25:31.526274 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:25:31.526238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:25:42.036475 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:25:42.036436    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:25:54.228406 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:25:54.228367    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:26:04.835970 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:04.835932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:26:21.237219 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:21.237184    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:26:34.027956 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:34.027919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:26:44.133972 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:44.133939    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:26:58.467238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.467209    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"]
Jul 14 01:26:58.470315 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.470299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.472903 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.472736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4928854b-serving-cert\""
Jul 14 01:26:58.472903 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.472777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:26:58.472903 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.472863    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:26:58.473455 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.473431    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:26:58.482610 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.482584    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"]
Jul 14 01:26:58.511317 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.511289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f305be-aaf7-4f01-82ca-a792d1b7e112-openshift-service-ca\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.511436 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.511331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.511436 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.511416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqbp\" (UniqueName: \"kubernetes.io/projected/17f305be-aaf7-4f01-82ca-a792d1b7e112-kube-api-access-mjqbp\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.612456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.612428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqbp\" (UniqueName: \"kubernetes.io/projected/17f305be-aaf7-4f01-82ca-a792d1b7e112-kube-api-access-mjqbp\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.612577 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.612480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f305be-aaf7-4f01-82ca-a792d1b7e112-openshift-service-ca\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.612577 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.612509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.614696 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.614665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4928854b-serving-cert\""
Jul 14 01:26:58.614696 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.614676    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:26:58.619932 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.619916    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:26:58.622710 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:26:58.622674    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-serving-cert: secret "maas-api-e2e-migrate-4928854b-serving-cert" not found
Jul 14 01:26:58.622795 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:26:58.622782    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls podName:17f305be-aaf7-4f01-82ca-a792d1b7e112 nodeName:}" failed. No retries permitted until 2026-07-14 01:26:59.122759833 +0000 UTC m=+1621.010024445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls") pod "maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" (UID: "17f305be-aaf7-4f01-82ca-a792d1b7e112") : secret "maas-api-e2e-migrate-4928854b-serving-cert" not found
Jul 14 01:26:58.623154 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.623135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f305be-aaf7-4f01-82ca-a792d1b7e112-openshift-service-ca\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:58.631448 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:58.631427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqbp\" (UniqueName: \"kubernetes.io/projected/17f305be-aaf7-4f01-82ca-a792d1b7e112-kube-api-access-mjqbp\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:59.218291 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.218248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:59.220797 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.220772    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls\") pod \"maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:59.390751 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.390722    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:26:59.399540 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.399510    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:26:59.533740 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.533713    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"]
Jul 14 01:26:59.535523 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:26:59.535495    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f305be_aaf7_4f01_82ca_a792d1b7e112.slice/crio-7fedba69319c9937179c843de84eeb4500995a4c89bd4cadcfddb10eddf59c1d WatchSource:0}: Error finding container 7fedba69319c9937179c843de84eeb4500995a4c89bd4cadcfddb10eddf59c1d: Status 404 returned error can't find the container with id 7fedba69319c9937179c843de84eeb4500995a4c89bd4cadcfddb10eddf59c1d
Jul 14 01:26:59.536804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.536785    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:26:59.951753 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:26:59.951716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" event={"ID":"17f305be-aaf7-4f01-82ca-a792d1b7e112","Type":"ContainerStarted","Data":"7fedba69319c9937179c843de84eeb4500995a4c89bd4cadcfddb10eddf59c1d"}
Jul 14 01:27:01.962518 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:27:01.962480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" event={"ID":"17f305be-aaf7-4f01-82ca-a792d1b7e112","Type":"ContainerStarted","Data":"04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4"}
Jul 14 01:27:01.962873 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:27:01.962612    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:27:01.979234 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:27:01.979190    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" podStartSLOduration=2.159103 podStartE2EDuration="3.97917688s" totalImagesPullingTime="1.82007388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:27:01.977789664 +0000 UTC m=+1623.865054293" watchObservedRunningTime="2026-07-14 01:27:01.97917688 +0000 UTC m=+1623.866441508"
Jul 14 01:27:06.729341 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:27:06.729306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:27:07.980754 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:27:07.980726    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:28:05.832767 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:05.832737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:28:08.878743 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:08.878698    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"]
Jul 14 01:28:08.879205 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:08.878913    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" podUID="17f305be-aaf7-4f01-82ca-a792d1b7e112" containerName="maas-api" containerID="cri-o://04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4" gracePeriod=30
Jul 14 01:28:09.117598 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.117576    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:28:09.244008 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.243919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjqbp\" (UniqueName: \"kubernetes.io/projected/17f305be-aaf7-4f01-82ca-a792d1b7e112-kube-api-access-mjqbp\") pod \"17f305be-aaf7-4f01-82ca-a792d1b7e112\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") "
Jul 14 01:28:09.244161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.244058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f305be-aaf7-4f01-82ca-a792d1b7e112-openshift-service-ca\") pod \"17f305be-aaf7-4f01-82ca-a792d1b7e112\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") "
Jul 14 01:28:09.244161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.244084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls\") pod \"17f305be-aaf7-4f01-82ca-a792d1b7e112\" (UID: \"17f305be-aaf7-4f01-82ca-a792d1b7e112\") "
Jul 14 01:28:09.244459 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.244436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f305be-aaf7-4f01-82ca-a792d1b7e112-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17f305be-aaf7-4f01-82ca-a792d1b7e112" (UID: "17f305be-aaf7-4f01-82ca-a792d1b7e112"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:28:09.246133 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.246111    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f305be-aaf7-4f01-82ca-a792d1b7e112-kube-api-access-mjqbp" (OuterVolumeSpecName: "kube-api-access-mjqbp") pod "17f305be-aaf7-4f01-82ca-a792d1b7e112" (UID: "17f305be-aaf7-4f01-82ca-a792d1b7e112"). InnerVolumeSpecName "kube-api-access-mjqbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:28:09.246209 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.246135    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17f305be-aaf7-4f01-82ca-a792d1b7e112" (UID: "17f305be-aaf7-4f01-82ca-a792d1b7e112"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:28:09.272507 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.272480    2578 generic.go:356] "Generic (PLEG): container finished" podID="17f305be-aaf7-4f01-82ca-a792d1b7e112" containerID="04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4" exitCode=0
Jul 14 01:28:09.272627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.272527    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"
Jul 14 01:28:09.272627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.272556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" event={"ID":"17f305be-aaf7-4f01-82ca-a792d1b7e112","Type":"ContainerDied","Data":"04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4"}
Jul 14 01:28:09.272627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.272609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl" event={"ID":"17f305be-aaf7-4f01-82ca-a792d1b7e112","Type":"ContainerDied","Data":"7fedba69319c9937179c843de84eeb4500995a4c89bd4cadcfddb10eddf59c1d"}
Jul 14 01:28:09.272804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.272630    2578 scope.go:117] "RemoveContainer" containerID="04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4"
Jul 14 01:28:09.281858 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.281841    2578 scope.go:117] "RemoveContainer" containerID="04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4"
Jul 14 01:28:09.282092 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:28:09.282074    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4\": container with ID starting with 04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4 not found: ID does not exist" containerID="04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4"
Jul 14 01:28:09.282153 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.282101    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4"} err="failed to get container status \"04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4\": rpc error: code = NotFound desc = could not find container \"04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4\": container with ID starting with 04c13621fd8b8b9a601031d0ab6da482f1a8c263ef49363e78e15d8cca3dcde4 not found: ID does not exist"
Jul 14 01:28:09.292649 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.292625    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"]
Jul 14 01:28:09.297049 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.297029    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4928854b-6679dbb858-8lwfl"]
Jul 14 01:28:09.345570 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.345547    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17f305be-aaf7-4f01-82ca-a792d1b7e112-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:28:09.345570 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.345568    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17f305be-aaf7-4f01-82ca-a792d1b7e112-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:28:09.345709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:09.345583    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjqbp\" (UniqueName: \"kubernetes.io/projected/17f305be-aaf7-4f01-82ca-a792d1b7e112-kube-api-access-mjqbp\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:28:10.690795 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:10.690762    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f305be-aaf7-4f01-82ca-a792d1b7e112" path="/var/lib/kubelet/pods/17f305be-aaf7-4f01-82ca-a792d1b7e112/volumes"
Jul 14 01:28:15.941667 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.941635    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"]
Jul 14 01:28:15.942082 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.942043    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f305be-aaf7-4f01-82ca-a792d1b7e112" containerName="maas-api"
Jul 14 01:28:15.942082 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.942056    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f305be-aaf7-4f01-82ca-a792d1b7e112" containerName="maas-api"
Jul 14 01:28:15.942167 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.942116    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f305be-aaf7-4f01-82ca-a792d1b7e112" containerName="maas-api"
Jul 14 01:28:15.946704 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.946670    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:15.948811 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.948779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e1db3cf4-serving-cert\""
Jul 14 01:28:15.948811 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.948804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:28:15.948967 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.948893    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:28:15.949759 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.949741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:28:15.955968 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:15.955946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"]
Jul 14 01:28:16.008345 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.008310    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khxjc\" (UniqueName: \"kubernetes.io/projected/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-kube-api-access-khxjc\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.008480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.008385    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.008480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.008404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-openshift-service-ca\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.109796 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.109766    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khxjc\" (UniqueName: \"kubernetes.io/projected/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-kube-api-access-khxjc\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.109995 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.109876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.109995 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.109904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-openshift-service-ca\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.110111 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:28:16.110004    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-serving-cert: secret "maas-api-e2e-ait-e1db3cf4-serving-cert" not found
Jul 14 01:28:16.110111 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:28:16.110089    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls podName:6c4eb224-4d1b-4c98-b484-c94c0f58d2b6 nodeName:}" failed. No retries permitted until 2026-07-14 01:28:16.610066906 +0000 UTC m=+1698.497331514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls") pod "maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" (UID: "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6") : secret "maas-api-e2e-ait-e1db3cf4-serving-cert" not found
Jul 14 01:28:16.110612 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.110589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-openshift-service-ca\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.123953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.123933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khxjc\" (UniqueName: \"kubernetes.io/projected/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-kube-api-access-khxjc\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.613476 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.613442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.616129 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.616106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls\") pod \"maas-api-e2e-ait-e1db3cf4-55486586f9-82chg\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.857853 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.857825    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:16.990098 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:16.990074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"]
Jul 14 01:28:16.992358 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:28:16.992325    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4eb224_4d1b_4c98_b484_c94c0f58d2b6.slice/crio-46eafdc1312bb52970986831d79da27c0fe7fb9fbf11430432c7d639d7de36e2 WatchSource:0}: Error finding container 46eafdc1312bb52970986831d79da27c0fe7fb9fbf11430432c7d639d7de36e2: Status 404 returned error can't find the container with id 46eafdc1312bb52970986831d79da27c0fe7fb9fbf11430432c7d639d7de36e2
Jul 14 01:28:17.312032 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:17.311999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" event={"ID":"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6","Type":"ContainerStarted","Data":"46eafdc1312bb52970986831d79da27c0fe7fb9fbf11430432c7d639d7de36e2"}
Jul 14 01:28:18.317883 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:18.317799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" event={"ID":"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6","Type":"ContainerStarted","Data":"0531e3aaa68ecd1a5bbf3bf0de2987f1afab1d25ae76f98418caa2b71eae1cf5"}
Jul 14 01:28:18.318226 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:18.317910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:28:18.335924 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:18.335855    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" podStartSLOduration=2.9212108199999998 podStartE2EDuration="3.33583582s" totalImagesPullingTime="414.625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:28:18.334678553 +0000 UTC m=+1700.221943181" watchObservedRunningTime="2026-07-14 01:28:18.33583582 +0000 UTC m=+1700.223100450"
Jul 14 01:28:19.429444 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:19.429369    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:28:24.327471 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:28:24.327443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:29:31.453286 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.453250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"]
Jul 14 01:29:31.453856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.453496    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" podUID="6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" containerName="maas-api" containerID="cri-o://0531e3aaa68ecd1a5bbf3bf0de2987f1afab1d25ae76f98418caa2b71eae1cf5" gracePeriod=30
Jul 14 01:29:31.667090 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.667060    2578 generic.go:356] "Generic (PLEG): container finished" podID="6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" containerID="0531e3aaa68ecd1a5bbf3bf0de2987f1afab1d25ae76f98418caa2b71eae1cf5" exitCode=0
Jul 14 01:29:31.667090 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.667095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" event={"ID":"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6","Type":"ContainerDied","Data":"0531e3aaa68ecd1a5bbf3bf0de2987f1afab1d25ae76f98418caa2b71eae1cf5"}
Jul 14 01:29:31.702284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.702263    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:29:31.754514 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.754451    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khxjc\" (UniqueName: \"kubernetes.io/projected/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-kube-api-access-khxjc\") pod \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") "
Jul 14 01:29:31.754514 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.754487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-openshift-service-ca\") pod \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") "
Jul 14 01:29:31.754726 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.754584    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls\") pod \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\" (UID: \"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6\") "
Jul 14 01:29:31.754887 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.754864    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" (UID: "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:29:31.754953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.754940    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:29:31.756605 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.756586    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-kube-api-access-khxjc" (OuterVolumeSpecName: "kube-api-access-khxjc") pod "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" (UID: "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6"). InnerVolumeSpecName "kube-api-access-khxjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:29:31.756739 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.756716    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" (UID: "6c4eb224-4d1b-4c98-b484-c94c0f58d2b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:29:31.856278 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.856249    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khxjc\" (UniqueName: \"kubernetes.io/projected/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-kube-api-access-khxjc\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:29:31.856278 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:31.856275    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:29:32.673044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:32.673019    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"
Jul 14 01:29:32.673443 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:32.673044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg" event={"ID":"6c4eb224-4d1b-4c98-b484-c94c0f58d2b6","Type":"ContainerDied","Data":"46eafdc1312bb52970986831d79da27c0fe7fb9fbf11430432c7d639d7de36e2"}
Jul 14 01:29:32.673443 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:32.673097    2578 scope.go:117] "RemoveContainer" containerID="0531e3aaa68ecd1a5bbf3bf0de2987f1afab1d25ae76f98418caa2b71eae1cf5"
Jul 14 01:29:32.693797 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:32.693770    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"]
Jul 14 01:29:32.695365 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:32.695347    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e1db3cf4-55486586f9-82chg"]
Jul 14 01:29:34.431591 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:34.431561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:29:34.690850 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:34.690779    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" path="/var/lib/kubelet/pods/6c4eb224-4d1b-4c98-b484-c94c0f58d2b6/volumes"
Jul 14 01:29:48.774305 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.774220    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"]
Jul 14 01:29:48.774826 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.774807    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" containerName="maas-api"
Jul 14 01:29:48.774877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.774830    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" containerName="maas-api"
Jul 14 01:29:48.774935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.774922    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4eb224-4d1b-4c98-b484-c94c0f58d2b6" containerName="maas-api"
Jul 14 01:29:48.779789 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.779769    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:48.782352 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.782330    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d3208c92-serving-cert\""
Jul 14 01:29:48.782464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.782384    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:29:48.782464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.782403    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:29:48.783091 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.783075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:29:48.787678 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.787644    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"]
Jul 14 01:29:48.906291 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.906263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:48.906456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.906307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eddc686-9441-42db-8f80-90a3189a9bd4-openshift-service-ca\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:48.906456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:48.906372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xh7\" (UniqueName: \"kubernetes.io/projected/5eddc686-9441-42db-8f80-90a3189a9bd4-kube-api-access-t6xh7\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.006864 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.006830    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eddc686-9441-42db-8f80-90a3189a9bd4-openshift-service-ca\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.007050 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.006912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xh7\" (UniqueName: \"kubernetes.io/projected/5eddc686-9441-42db-8f80-90a3189a9bd4-kube-api-access-t6xh7\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.007050 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.006978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.007172 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:29:49.007088    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-serving-cert: secret "maas-api-e2e-derive-d3208c92-serving-cert" not found
Jul 14 01:29:49.007172 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:29:49.007160    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls podName:5eddc686-9441-42db-8f80-90a3189a9bd4 nodeName:}" failed. No retries permitted until 2026-07-14 01:29:49.507138493 +0000 UTC m=+1791.394403101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls") pod "maas-api-e2e-derive-d3208c92-f579db765-sxfnl" (UID: "5eddc686-9441-42db-8f80-90a3189a9bd4") : secret "maas-api-e2e-derive-d3208c92-serving-cert" not found
Jul 14 01:29:49.007590 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.007528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eddc686-9441-42db-8f80-90a3189a9bd4-openshift-service-ca\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.019044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.019018    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xh7\" (UniqueName: \"kubernetes.io/projected/5eddc686-9441-42db-8f80-90a3189a9bd4-kube-api-access-t6xh7\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.511653 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.511619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.514112 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.514083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls\") pod \"maas-api-e2e-derive-d3208c92-f579db765-sxfnl\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.691294 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.691263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:49.827851 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:49.827827    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"]
Jul 14 01:29:49.830083 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:29:49.830059    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eddc686_9441_42db_8f80_90a3189a9bd4.slice/crio-9776f160392770f980053f63510ba91738f01eebf3e94f52710751ace41f96fd WatchSource:0}: Error finding container 9776f160392770f980053f63510ba91738f01eebf3e94f52710751ace41f96fd: Status 404 returned error can't find the container with id 9776f160392770f980053f63510ba91738f01eebf3e94f52710751ace41f96fd
Jul 14 01:29:50.760265 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:50.760231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl" event={"ID":"5eddc686-9441-42db-8f80-90a3189a9bd4","Type":"ContainerStarted","Data":"3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b"}
Jul 14 01:29:50.760469 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:50.760283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl" event={"ID":"5eddc686-9441-42db-8f80-90a3189a9bd4","Type":"ContainerStarted","Data":"9776f160392770f980053f63510ba91738f01eebf3e94f52710751ace41f96fd"}
Jul 14 01:29:50.760469 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:50.760356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:50.777488 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:50.777441    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl" podStartSLOduration=2.292642131 podStartE2EDuration="2.777426206s" totalImagesPullingTime="484.784075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:29:50.775449667 +0000 UTC m=+1792.662714296" watchObservedRunningTime="2026-07-14 01:29:50.777426206 +0000 UTC m=+1792.664690834"
Jul 14 01:29:51.232431 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:51.232396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:29:56.770106 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:56.770080    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:29:58.674179 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:58.674145    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:29:58.680220 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:29:58.680194    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:30:00.131803 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.131773    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"]
Jul 14 01:30:00.135283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.135267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"
Jul 14 01:30:00.137377 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.137356    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zn7l8\""
Jul 14 01:30:00.141624 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.141602    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"]
Jul 14 01:30:00.146794 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.146775    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"
Jul 14 01:30:00.277013 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.276985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"]
Jul 14 01:30:00.279743 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:30:00.279718    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ea1579_940b_4f18_a1cf_3f5c885d7e59.slice/crio-e62d073ff5afe88dee93d2cc0e9bee1213da00c8da6e443ff9d7e3d47e275533 WatchSource:0}: Error finding container e62d073ff5afe88dee93d2cc0e9bee1213da00c8da6e443ff9d7e3d47e275533: Status 404 returned error can't find the container with id e62d073ff5afe88dee93d2cc0e9bee1213da00c8da6e443ff9d7e3d47e275533
Jul 14 01:30:00.811480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:00.811447    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb" event={"ID":"19ea1579-940b-4f18-a1cf-3f5c885d7e59","Type":"ContainerStarted","Data":"e62d073ff5afe88dee93d2cc0e9bee1213da00c8da6e443ff9d7e3d47e275533"}
Jul 14 01:30:03.830125 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:03.830086    2578 generic.go:356] "Generic (PLEG): container finished" podID="19ea1579-940b-4f18-a1cf-3f5c885d7e59" containerID="faa4eb9337cd50e4a90d0cf6c556f17f86449b6fbd6f4ba6a1a5bb6e324e5768" exitCode=0
Jul 14 01:30:03.830546 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:03.830169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb" event={"ID":"19ea1579-940b-4f18-a1cf-3f5c885d7e59","Type":"ContainerDied","Data":"faa4eb9337cd50e4a90d0cf6c556f17f86449b6fbd6f4ba6a1a5bb6e324e5768"}
Jul 14 01:30:04.963738 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:04.963715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"
Jul 14 01:30:05.840837 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:05.840801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb" event={"ID":"19ea1579-940b-4f18-a1cf-3f5c885d7e59","Type":"ContainerDied","Data":"e62d073ff5afe88dee93d2cc0e9bee1213da00c8da6e443ff9d7e3d47e275533"}
Jul 14 01:30:05.840837 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:05.840836    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"
Jul 14 01:30:05.841038 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:05.840853    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e62d073ff5afe88dee93d2cc0e9bee1213da00c8da6e443ff9d7e3d47e275533"
Jul 14 01:30:43.137800 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.137765    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"]
Jul 14 01:30:43.140093 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.138138    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19ea1579-940b-4f18-a1cf-3f5c885d7e59" containerName="cleanup"
Jul 14 01:30:43.140093 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.138148    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19ea1579-940b-4f18-a1cf-3f5c885d7e59" containerName="cleanup"
Jul 14 01:30:43.140093 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.138222    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19ea1579-940b-4f18-a1cf-3f5c885d7e59" containerName="cleanup"
Jul 14 01:30:43.141071 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.141054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:43.143061 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.143036    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zn7l8\""
Jul 14 01:30:43.146856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.146834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"]
Jul 14 01:30:43.273643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.273613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb734f39-eb62-48d9-99e1-4df4ff524181-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d3208c92-dg49w\" (UID: \"eb734f39-eb62-48d9-99e1-4df4ff524181\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:43.374454 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.374421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb734f39-eb62-48d9-99e1-4df4ff524181-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d3208c92-dg49w\" (UID: \"eb734f39-eb62-48d9-99e1-4df4ff524181\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:43.375050 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.375034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb734f39-eb62-48d9-99e1-4df4ff524181-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-d3208c92-dg49w\" (UID: \"eb734f39-eb62-48d9-99e1-4df4ff524181\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:43.453133 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.453095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:43.589182 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:43.589154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"]
Jul 14 01:30:43.591331 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:30:43.591305    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb734f39_eb62_48d9_99e1_4df4ff524181.slice/crio-881dca9ad64c899a28367c8fcb79c1bcced7fbdf6f63cde6fd5c7318c0cd16c1 WatchSource:0}: Error finding container 881dca9ad64c899a28367c8fcb79c1bcced7fbdf6f63cde6fd5c7318c0cd16c1: Status 404 returned error can't find the container with id 881dca9ad64c899a28367c8fcb79c1bcced7fbdf6f63cde6fd5c7318c0cd16c1
Jul 14 01:30:44.016759 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:44.016722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w" event={"ID":"eb734f39-eb62-48d9-99e1-4df4ff524181","Type":"ContainerStarted","Data":"7e5208314ab89e1398a8b89a5a7b016148e4d49a037c569a72c791b1240e4232"}
Jul 14 01:30:44.016921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:44.016772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w" event={"ID":"eb734f39-eb62-48d9-99e1-4df4ff524181","Type":"ContainerStarted","Data":"881dca9ad64c899a28367c8fcb79c1bcced7fbdf6f63cde6fd5c7318c0cd16c1"}
Jul 14 01:30:44.032049 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:44.031960    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w" podStartSLOduration=1.031946449 podStartE2EDuration="1.031946449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:30:44.031469975 +0000 UTC m=+1845.918734604" watchObservedRunningTime="2026-07-14 01:30:44.031946449 +0000 UTC m=+1845.919211078"
Jul 14 01:30:45.023309 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:45.023271    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb734f39-eb62-48d9-99e1-4df4ff524181" containerID="7e5208314ab89e1398a8b89a5a7b016148e4d49a037c569a72c791b1240e4232" exitCode=0
Jul 14 01:30:45.023721 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:45.023361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w" event={"ID":"eb734f39-eb62-48d9-99e1-4df4ff524181","Type":"ContainerDied","Data":"7e5208314ab89e1398a8b89a5a7b016148e4d49a037c569a72c791b1240e4232"}
Jul 14 01:30:46.155443 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:46.155417    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:46.297645 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:46.297564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb734f39-eb62-48d9-99e1-4df4ff524181-maas-api-service-ca\") pod \"eb734f39-eb62-48d9-99e1-4df4ff524181\" (UID: \"eb734f39-eb62-48d9-99e1-4df4ff524181\") "
Jul 14 01:30:46.297931 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:46.297909    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb734f39-eb62-48d9-99e1-4df4ff524181-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb734f39-eb62-48d9-99e1-4df4ff524181" (UID: "eb734f39-eb62-48d9-99e1-4df4ff524181"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:30:46.398567 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:46.398543    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb734f39-eb62-48d9-99e1-4df4ff524181-maas-api-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:30:47.033473 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:47.033396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w" event={"ID":"eb734f39-eb62-48d9-99e1-4df4ff524181","Type":"ContainerDied","Data":"881dca9ad64c899a28367c8fcb79c1bcced7fbdf6f63cde6fd5c7318c0cd16c1"}
Jul 14 01:30:47.033473 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:47.033431    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d3208c92-dg49w"
Jul 14 01:30:47.033473 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:47.033447    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881dca9ad64c899a28367c8fcb79c1bcced7fbdf6f63cde6fd5c7318c0cd16c1"
Jul 14 01:30:53.618203 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.618169    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"]
Jul 14 01:30:53.618708 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.618424    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl" podUID="5eddc686-9441-42db-8f80-90a3189a9bd4" containerName="maas-api" containerID="cri-o://3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b" gracePeriod=30
Jul 14 01:30:53.644923 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.644892    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"]
Jul 14 01:30:53.648211 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.648186    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-d3208c92-29733210-7fxnb"]
Jul 14 01:30:53.861587 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.861565    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:30:53.958918 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.958890    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls\") pod \"5eddc686-9441-42db-8f80-90a3189a9bd4\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") "
Jul 14 01:30:53.959108 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.958972    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eddc686-9441-42db-8f80-90a3189a9bd4-openshift-service-ca\") pod \"5eddc686-9441-42db-8f80-90a3189a9bd4\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") "
Jul 14 01:30:53.959108 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.959013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6xh7\" (UniqueName: \"kubernetes.io/projected/5eddc686-9441-42db-8f80-90a3189a9bd4-kube-api-access-t6xh7\") pod \"5eddc686-9441-42db-8f80-90a3189a9bd4\" (UID: \"5eddc686-9441-42db-8f80-90a3189a9bd4\") "
Jul 14 01:30:53.959384 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.959361    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eddc686-9441-42db-8f80-90a3189a9bd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5eddc686-9441-42db-8f80-90a3189a9bd4" (UID: "5eddc686-9441-42db-8f80-90a3189a9bd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:30:53.961091 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.961061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5eddc686-9441-42db-8f80-90a3189a9bd4" (UID: "5eddc686-9441-42db-8f80-90a3189a9bd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:30:53.961456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:53.961430    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eddc686-9441-42db-8f80-90a3189a9bd4-kube-api-access-t6xh7" (OuterVolumeSpecName: "kube-api-access-t6xh7") pod "5eddc686-9441-42db-8f80-90a3189a9bd4" (UID: "5eddc686-9441-42db-8f80-90a3189a9bd4"). InnerVolumeSpecName "kube-api-access-t6xh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:30:54.060401 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.060376    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eddc686-9441-42db-8f80-90a3189a9bd4-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:30:54.060401 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.060402    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xh7\" (UniqueName: \"kubernetes.io/projected/5eddc686-9441-42db-8f80-90a3189a9bd4-kube-api-access-t6xh7\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:30:54.060524 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.060412    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5eddc686-9441-42db-8f80-90a3189a9bd4-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:30:54.066580 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.066552    2578 generic.go:356] "Generic (PLEG): container finished" podID="5eddc686-9441-42db-8f80-90a3189a9bd4" containerID="3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b" exitCode=0
Jul 14 01:30:54.066680 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.066599    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"
Jul 14 01:30:54.066680 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.066602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl" event={"ID":"5eddc686-9441-42db-8f80-90a3189a9bd4","Type":"ContainerDied","Data":"3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b"}
Jul 14 01:30:54.066822 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.066744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl" event={"ID":"5eddc686-9441-42db-8f80-90a3189a9bd4","Type":"ContainerDied","Data":"9776f160392770f980053f63510ba91738f01eebf3e94f52710751ace41f96fd"}
Jul 14 01:30:54.066822 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.066773    2578 scope.go:117] "RemoveContainer" containerID="3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b"
Jul 14 01:30:54.075236 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.075222    2578 scope.go:117] "RemoveContainer" containerID="3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b"
Jul 14 01:30:54.075503 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:30:54.075488    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b\": container with ID starting with 3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b not found: ID does not exist" containerID="3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b"
Jul 14 01:30:54.075552 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.075510    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b"} err="failed to get container status \"3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b\": rpc error: code = NotFound desc = could not find container \"3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b\": container with ID starting with 3fd9c7796663cc5e745f862841d1bf49d55eb76e858fb141bf74ddb4c62c673b not found: ID does not exist"
Jul 14 01:30:54.088176 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.088156    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"]
Jul 14 01:30:54.091763 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.091745    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d3208c92-f579db765-sxfnl"]
Jul 14 01:30:54.691779 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.691748    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19ea1579-940b-4f18-a1cf-3f5c885d7e59" path="/var/lib/kubelet/pods/19ea1579-940b-4f18-a1cf-3f5c885d7e59/volumes"
Jul 14 01:30:54.692152 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:54.692112    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eddc686-9441-42db-8f80-90a3189a9bd4" path="/var/lib/kubelet/pods/5eddc686-9441-42db-8f80-90a3189a9bd4/volumes"
Jul 14 01:30:56.429931 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:30:56.429899    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:32:30.330518 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:30.330482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:32:39.329038 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:39.329002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:32:46.431640 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.431600    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"]
Jul 14 01:32:46.432735 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.432710    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb734f39-eb62-48d9-99e1-4df4ff524181" containerName="revoke-keys"
Jul 14 01:32:46.432865 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.432855    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb734f39-eb62-48d9-99e1-4df4ff524181" containerName="revoke-keys"
Jul 14 01:32:46.432989 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.432965    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eddc686-9441-42db-8f80-90a3189a9bd4" containerName="maas-api"
Jul 14 01:32:46.432989 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.432983    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eddc686-9441-42db-8f80-90a3189a9bd4" containerName="maas-api"
Jul 14 01:32:46.433163 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.433084    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb734f39-eb62-48d9-99e1-4df4ff524181" containerName="revoke-keys"
Jul 14 01:32:46.433163 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.433098    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eddc686-9441-42db-8f80-90a3189a9bd4" containerName="maas-api"
Jul 14 01:32:46.436409 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.436389    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.438886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.438844    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-40103c45-serving-cert\""
Jul 14 01:32:46.439063 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.438905    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:32:46.440727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.439936    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:32:46.440727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.440007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:32:46.442378 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.442345    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"]
Jul 14 01:32:46.526396 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.526367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edddc609-6259-4b01-ba40-3a5ed867927b-openshift-service-ca\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.526576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.526408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edddc609-6259-4b01-ba40-3a5ed867927b-maas-api-tls\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.526576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.526441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm94l\" (UniqueName: \"kubernetes.io/projected/edddc609-6259-4b01-ba40-3a5ed867927b-kube-api-access-nm94l\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.627893 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.627862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edddc609-6259-4b01-ba40-3a5ed867927b-openshift-service-ca\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.628052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.627904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edddc609-6259-4b01-ba40-3a5ed867927b-maas-api-tls\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.628052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.627951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm94l\" (UniqueName: \"kubernetes.io/projected/edddc609-6259-4b01-ba40-3a5ed867927b-kube-api-access-nm94l\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.628590 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.628560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edddc609-6259-4b01-ba40-3a5ed867927b-openshift-service-ca\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.630500 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.630479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edddc609-6259-4b01-ba40-3a5ed867927b-maas-api-tls\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.635098 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.635079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm94l\" (UniqueName: \"kubernetes.io/projected/edddc609-6259-4b01-ba40-3a5ed867927b-kube-api-access-nm94l\") pod \"maas-api-e2e-mt-40103c45-895677c58-dht5g\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.748153 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.748083    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:46.880653 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.880629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"]
Jul 14 01:32:46.883428 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:32:46.883406    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedddc609_6259_4b01_ba40_3a5ed867927b.slice/crio-117e80716b1c1461ed01c63bb4957b6682cffac30f2dd2960a7005c217011256 WatchSource:0}: Error finding container 117e80716b1c1461ed01c63bb4957b6682cffac30f2dd2960a7005c217011256: Status 404 returned error can't find the container with id 117e80716b1c1461ed01c63bb4957b6682cffac30f2dd2960a7005c217011256
Jul 14 01:32:46.885152 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:46.885135    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:32:47.602721 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:47.602618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" event={"ID":"edddc609-6259-4b01-ba40-3a5ed867927b","Type":"ContainerStarted","Data":"b3812189208bdf4a6c80095d55a66b54aa4625a8280b28c3e26257eb967eff3e"}
Jul 14 01:32:47.602721 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:47.602680    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" event={"ID":"edddc609-6259-4b01-ba40-3a5ed867927b","Type":"ContainerStarted","Data":"117e80716b1c1461ed01c63bb4957b6682cffac30f2dd2960a7005c217011256"}
Jul 14 01:32:47.603215 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:47.602737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:32:47.619105 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:47.619053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" podStartSLOduration=1.248541956 podStartE2EDuration="1.619037914s" totalImagesPullingTime="370.495958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:32:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:32:47.617288116 +0000 UTC m=+1969.504552745" watchObservedRunningTime="2026-07-14 01:32:47.619037914 +0000 UTC m=+1969.506302523"
Jul 14 01:32:52.032809 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:52.032734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:32:53.611935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:32:53.611861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:33:08.496510 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.496478    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"]
Jul 14 01:33:08.497118 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.496714    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" podUID="edddc609-6259-4b01-ba40-3a5ed867927b" containerName="maas-api" containerID="cri-o://b3812189208bdf4a6c80095d55a66b54aa4625a8280b28c3e26257eb967eff3e" gracePeriod=30
Jul 14 01:33:08.607876 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.607818    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" podUID="edddc609-6259-4b01-ba40-3a5ed867927b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.33:8443/health\": dial tcp 10.134.0.33:8443: connect: connection refused"
Jul 14 01:33:08.702143 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.702110    2578 generic.go:356] "Generic (PLEG): container finished" podID="edddc609-6259-4b01-ba40-3a5ed867927b" containerID="b3812189208bdf4a6c80095d55a66b54aa4625a8280b28c3e26257eb967eff3e" exitCode=0
Jul 14 01:33:08.702295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.702178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" event={"ID":"edddc609-6259-4b01-ba40-3a5ed867927b","Type":"ContainerDied","Data":"b3812189208bdf4a6c80095d55a66b54aa4625a8280b28c3e26257eb967eff3e"}
Jul 14 01:33:08.742780 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.742753    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:33:08.922023 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.921995    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edddc609-6259-4b01-ba40-3a5ed867927b-maas-api-tls\") pod \"edddc609-6259-4b01-ba40-3a5ed867927b\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") "
Jul 14 01:33:08.922210 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.922071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edddc609-6259-4b01-ba40-3a5ed867927b-openshift-service-ca\") pod \"edddc609-6259-4b01-ba40-3a5ed867927b\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") "
Jul 14 01:33:08.922210 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.922136    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm94l\" (UniqueName: \"kubernetes.io/projected/edddc609-6259-4b01-ba40-3a5ed867927b-kube-api-access-nm94l\") pod \"edddc609-6259-4b01-ba40-3a5ed867927b\" (UID: \"edddc609-6259-4b01-ba40-3a5ed867927b\") "
Jul 14 01:33:08.922428 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.922401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edddc609-6259-4b01-ba40-3a5ed867927b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edddc609-6259-4b01-ba40-3a5ed867927b" (UID: "edddc609-6259-4b01-ba40-3a5ed867927b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:33:08.924417 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.924383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edddc609-6259-4b01-ba40-3a5ed867927b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edddc609-6259-4b01-ba40-3a5ed867927b" (UID: "edddc609-6259-4b01-ba40-3a5ed867927b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:33:08.924508 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:08.924479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddc609-6259-4b01-ba40-3a5ed867927b-kube-api-access-nm94l" (OuterVolumeSpecName: "kube-api-access-nm94l") pod "edddc609-6259-4b01-ba40-3a5ed867927b" (UID: "edddc609-6259-4b01-ba40-3a5ed867927b"). InnerVolumeSpecName "kube-api-access-nm94l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:33:09.023436 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.023399    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edddc609-6259-4b01-ba40-3a5ed867927b-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:33:09.023436 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.023431    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edddc609-6259-4b01-ba40-3a5ed867927b-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:33:09.023436 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.023441    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm94l\" (UniqueName: \"kubernetes.io/projected/edddc609-6259-4b01-ba40-3a5ed867927b-kube-api-access-nm94l\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:33:09.708495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.708468    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"
Jul 14 01:33:09.708495 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.708489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g" event={"ID":"edddc609-6259-4b01-ba40-3a5ed867927b","Type":"ContainerDied","Data":"117e80716b1c1461ed01c63bb4957b6682cffac30f2dd2960a7005c217011256"}
Jul 14 01:33:09.709034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.708553    2578 scope.go:117] "RemoveContainer" containerID="b3812189208bdf4a6c80095d55a66b54aa4625a8280b28c3e26257eb967eff3e"
Jul 14 01:33:09.728570 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.728548    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"]
Jul 14 01:33:09.732476 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:09.732454    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40103c45-895677c58-dht5g"]
Jul 14 01:33:10.690884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:10.690844    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edddc609-6259-4b01-ba40-3a5ed867927b" path="/var/lib/kubelet/pods/edddc609-6259-4b01-ba40-3a5ed867927b/volumes"
Jul 14 01:33:11.330782 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:11.330750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:33:45.750477 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.750439    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"]
Jul 14 01:33:45.751032 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.751014    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edddc609-6259-4b01-ba40-3a5ed867927b" containerName="maas-api"
Jul 14 01:33:45.751081 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.751035    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="edddc609-6259-4b01-ba40-3a5ed867927b" containerName="maas-api"
Jul 14 01:33:45.751147 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.751136    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="edddc609-6259-4b01-ba40-3a5ed867927b" containerName="maas-api"
Jul 14 01:33:45.754615 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.754598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.756979 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.756957    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:33:45.757758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.757735    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-45ee56e2-serving-cert\""
Jul 14 01:33:45.757872 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.757735    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:33:45.757872 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.757745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:33:45.762621 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.762599    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"]
Jul 14 01:33:45.828239 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.828206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-openshift-service-ca\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.828414 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.828247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-maas-api-tls\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.828414 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.828324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkwv\" (UniqueName: \"kubernetes.io/projected/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-kube-api-access-qjkwv\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.929145 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.929115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-openshift-service-ca\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.929145 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.929148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-maas-api-tls\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.929336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.929181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkwv\" (UniqueName: \"kubernetes.io/projected/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-kube-api-access-qjkwv\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.929755 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.929727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-openshift-service-ca\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.931695 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.931663    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-maas-api-tls\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:45.937999 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:45.937975    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkwv\" (UniqueName: \"kubernetes.io/projected/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-kube-api-access-qjkwv\") pod \"maas-api-e2e-mt-45ee56e2-86f55d8799-54tns\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:46.066850 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:46.066777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:46.198339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:46.198290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"]
Jul 14 01:33:46.200625 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:33:46.200592    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f5fc51_78e7_430a_beb9_6f7d3f2a2466.slice/crio-39d63cb1de72768b96d6b7a4480e26f78f9064bc6a32fec6236c60676bdb7f72 WatchSource:0}: Error finding container 39d63cb1de72768b96d6b7a4480e26f78f9064bc6a32fec6236c60676bdb7f72: Status 404 returned error can't find the container with id 39d63cb1de72768b96d6b7a4480e26f78f9064bc6a32fec6236c60676bdb7f72
Jul 14 01:33:46.881245 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:46.881207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns" event={"ID":"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466","Type":"ContainerStarted","Data":"d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0"}
Jul 14 01:33:46.881707 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:46.881272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns" event={"ID":"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466","Type":"ContainerStarted","Data":"39d63cb1de72768b96d6b7a4480e26f78f9064bc6a32fec6236c60676bdb7f72"}
Jul 14 01:33:46.881707 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:46.881306    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:33:46.898209 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:46.898165    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns" podStartSLOduration=1.403487747 podStartE2EDuration="1.898147799s" totalImagesPullingTime="494.660052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:33:46.896638981 +0000 UTC m=+2028.783903609" watchObservedRunningTime="2026-07-14 01:33:46.898147799 +0000 UTC m=+2028.785412426"
Jul 14 01:33:48.429954 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:48.429916    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:33:52.891978 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:33:52.891948    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:34:19.933454 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.933373    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"]
Jul 14 01:34:19.936944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.936925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:19.939293 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.939270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-45ee56e2\"/\"kube-root-ca.crt\""
Jul 14 01:34:19.940052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.940021    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-45ee56e2\"/\"openshift-service-ca.crt\""
Jul 14 01:34:19.940052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.940038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-45ee56e2\"/\"e2e-lifecycle-model-45ee56e2-kserve-self-signed-certs\""
Jul 14 01:34:19.940216 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.940042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-45ee56e2\"/\"default-dockercfg-cwpzf\""
Jul 14 01:34:19.943520 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:19.943496    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"]
Jul 14 01:34:20.018964 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.018933    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2s4\" (UniqueName: \"kubernetes.io/projected/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-kube-api-access-nd2s4\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.019113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.018997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tls-certs\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.019113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.019021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tmp-dir\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.019113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.019067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-model-cache\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.019113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.019096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-home\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.019113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.019109    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-dshm\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119517 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tls-certs\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119517 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tmp-dir\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119534    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-model-cache\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-home\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-dshm\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2s4\" (UniqueName: \"kubernetes.io/projected/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-kube-api-access-nd2s4\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.119975 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tmp-dir\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.120028 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.119980    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-home\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.120028 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.120010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-model-cache\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.122034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.122004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-dshm\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.122243 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.122227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tls-certs\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.127336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.127311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2s4\" (UniqueName: \"kubernetes.io/projected/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-kube-api-access-nd2s4\") pod \"e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") " pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.247758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.247664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:20.379894 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:20.379865    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"]
Jul 14 01:34:20.381806 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:34:20.381781    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf9e2168_2c0f_4f7a_8a69_0c58293ad775.slice/crio-2bccc43c51ff5ccfdd506fa82050de370dd583c526f043e0c6a83e2540c7f97c WatchSource:0}: Error finding container 2bccc43c51ff5ccfdd506fa82050de370dd583c526f043e0c6a83e2540c7f97c: Status 404 returned error can't find the container with id 2bccc43c51ff5ccfdd506fa82050de370dd583c526f043e0c6a83e2540c7f97c
Jul 14 01:34:21.039632 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:21.039595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z" event={"ID":"bf9e2168-2c0f-4f7a-8a69-0c58293ad775","Type":"ContainerStarted","Data":"39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a"}
Jul 14 01:34:21.039632 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:21.039636    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z" event={"ID":"bf9e2168-2c0f-4f7a-8a69-0c58293ad775","Type":"ContainerStarted","Data":"2bccc43c51ff5ccfdd506fa82050de370dd583c526f043e0c6a83e2540c7f97c"}
Jul 14 01:34:21.056117 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:21.056073    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z" podStartSLOduration=2.056054277 podStartE2EDuration="2.056054277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:34:21.054888069 +0000 UTC m=+2062.942152702" watchObservedRunningTime="2026-07-14 01:34:21.056054277 +0000 UTC m=+2062.943318907"
Jul 14 01:34:23.135303 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:23.135265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:34:30.248439 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:30.248382    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:30.248903 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:30.248460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:30.260959 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:30.260933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:31.101637 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:31.101611    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:34:35.935553 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:35.935515    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:34:49.762789 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:49.762757    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"]
Jul 14 01:34:49.763207 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:49.762948    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns" podUID="f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" containerName="maas-api" containerID="cri-o://d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0" gracePeriod=30
Jul 14 01:34:50.007995 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.007970    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:34:50.175707 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.175660    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" containerID="d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0" exitCode=0
Jul 14 01:34:50.175854 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.175733    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"
Jul 14 01:34:50.175854 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.175745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns" event={"ID":"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466","Type":"ContainerDied","Data":"d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0"}
Jul 14 01:34:50.175854 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.175824    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns" event={"ID":"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466","Type":"ContainerDied","Data":"39d63cb1de72768b96d6b7a4480e26f78f9064bc6a32fec6236c60676bdb7f72"}
Jul 14 01:34:50.175854 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.175847    2578 scope.go:117] "RemoveContainer" containerID="d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0"
Jul 14 01:34:50.184030 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.184014    2578 scope.go:117] "RemoveContainer" containerID="d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0"
Jul 14 01:34:50.184290 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:34:50.184274    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0\": container with ID starting with d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0 not found: ID does not exist" containerID="d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0"
Jul 14 01:34:50.184336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.184296    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0"} err="failed to get container status \"d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0\": rpc error: code = NotFound desc = could not find container \"d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0\": container with ID starting with d07004b81a6860a645d90904763b9549be69c13d50fe91176e21e2014af5bdb0 not found: ID does not exist"
Jul 14 01:34:50.186621 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.186604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-openshift-service-ca\") pod \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") "
Jul 14 01:34:50.186670 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.186651    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-maas-api-tls\") pod \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") "
Jul 14 01:34:50.186746 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.186711    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjkwv\" (UniqueName: \"kubernetes.io/projected/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-kube-api-access-qjkwv\") pod \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\" (UID: \"f3f5fc51-78e7-430a-beb9-6f7d3f2a2466\") "
Jul 14 01:34:50.186950 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.186929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" (UID: "f3f5fc51-78e7-430a-beb9-6f7d3f2a2466"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:34:50.188860 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.188837    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" (UID: "f3f5fc51-78e7-430a-beb9-6f7d3f2a2466"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:34:50.188946 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.188856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-kube-api-access-qjkwv" (OuterVolumeSpecName: "kube-api-access-qjkwv") pod "f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" (UID: "f3f5fc51-78e7-430a-beb9-6f7d3f2a2466"). InnerVolumeSpecName "kube-api-access-qjkwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:50.287644 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.287617    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:34:50.287644 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.287641    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjkwv\" (UniqueName: \"kubernetes.io/projected/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-kube-api-access-qjkwv\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:34:50.287644 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.287651    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:34:50.496558 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.496524    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"]
Jul 14 01:34:50.498254 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.498233    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-45ee56e2-86f55d8799-54tns"]
Jul 14 01:34:50.691930 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:50.691895    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" path="/var/lib/kubelet/pods/f3f5fc51-78e7-430a-beb9-6f7d3f2a2466/volumes"
Jul 14 01:34:53.729006 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:53.728967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:34:58.704383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:58.704355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:34:58.712116 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:58.712095    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:34:59.038416 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:34:59.038334    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"]
Jul 14 01:35:14.060347 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.060301    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z" podUID="bf9e2168-2c0f-4f7a-8a69-0c58293ad775" containerName="main" containerID="cri-o://39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a" gracePeriod=45
Jul 14 01:35:14.084917 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:35:14.084885    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf9e2168_2c0f_4f7a_8a69_0c58293ad775.slice/crio-conmon-39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf9e2168_2c0f_4f7a_8a69_0c58293ad775.slice/crio-39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a.scope\": RecentStats: unable to find data in memory cache]"
Jul 14 01:35:14.293003 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.292971    2578 generic.go:356] "Generic (PLEG): container finished" podID="bf9e2168-2c0f-4f7a-8a69-0c58293ad775" containerID="39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a" exitCode=0
Jul 14 01:35:14.293175 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.293040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z" event={"ID":"bf9e2168-2c0f-4f7a-8a69-0c58293ad775","Type":"ContainerDied","Data":"39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a"}
Jul 14 01:35:14.316677 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.316622    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:35:14.399041 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tls-certs\") pod \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") "
Jul 14 01:35:14.399041 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399049    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-model-cache\") pod \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") "
Jul 14 01:35:14.399283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-dshm\") pod \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") "
Jul 14 01:35:14.399283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tmp-dir\") pod \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") "
Jul 14 01:35:14.399283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-home\") pod \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") "
Jul 14 01:35:14.399283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd2s4\" (UniqueName: \"kubernetes.io/projected/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-kube-api-access-nd2s4\") pod \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\" (UID: \"bf9e2168-2c0f-4f7a-8a69-0c58293ad775\") "
Jul 14 01:35:14.399483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf9e2168-2c0f-4f7a-8a69-0c58293ad775" (UID: "bf9e2168-2c0f-4f7a-8a69-0c58293ad775"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:35:14.399483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf9e2168-2c0f-4f7a-8a69-0c58293ad775" (UID: "bf9e2168-2c0f-4f7a-8a69-0c58293ad775"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:35:14.399483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-home" (OuterVolumeSpecName: "home") pod "bf9e2168-2c0f-4f7a-8a69-0c58293ad775" (UID: "bf9e2168-2c0f-4f7a-8a69-0c58293ad775"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:35:14.399628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399532    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:35:14.399628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399545    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:35:14.399628 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.399554    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:35:14.401665 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.401641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-kube-api-access-nd2s4" (OuterVolumeSpecName: "kube-api-access-nd2s4") pod "bf9e2168-2c0f-4f7a-8a69-0c58293ad775" (UID: "bf9e2168-2c0f-4f7a-8a69-0c58293ad775"). InnerVolumeSpecName "kube-api-access-nd2s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:35:14.401793 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.401770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf9e2168-2c0f-4f7a-8a69-0c58293ad775" (UID: "bf9e2168-2c0f-4f7a-8a69-0c58293ad775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:35:14.401793 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.401782    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-dshm" (OuterVolumeSpecName: "dshm") pod "bf9e2168-2c0f-4f7a-8a69-0c58293ad775" (UID: "bf9e2168-2c0f-4f7a-8a69-0c58293ad775"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:35:14.500630 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.500594    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2s4\" (UniqueName: \"kubernetes.io/projected/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-kube-api-access-nd2s4\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:35:14.500630 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.500624    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:35:14.500630 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:14.500634    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf9e2168-2c0f-4f7a-8a69-0c58293ad775-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:35:15.299496 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:15.299464    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z" event={"ID":"bf9e2168-2c0f-4f7a-8a69-0c58293ad775","Type":"ContainerDied","Data":"2bccc43c51ff5ccfdd506fa82050de370dd583c526f043e0c6a83e2540c7f97c"}
Jul 14 01:35:15.299496 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:15.299498    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"
Jul 14 01:35:15.300019 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:15.299516    2578 scope.go:117] "RemoveContainer" containerID="39b1388a91d5d258c7282e25d3cf8b30b10cb36aafd31611c8dd380003aa6e6a"
Jul 14 01:35:15.314630 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:15.314604    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"]
Jul 14 01:35:15.318283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:15.318261    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-45ee56e2/e2e-lifecycle-model-45ee56e2-kserve-676878b87-zrl8z"]
Jul 14 01:35:16.690555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:16.690523    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf9e2168-2c0f-4f7a-8a69-0c58293ad775" path="/var/lib/kubelet/pods/bf9e2168-2c0f-4f7a-8a69-0c58293ad775/volumes"
Jul 14 01:35:50.729163 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:35:50.729088    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:36:04.929932 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:04.929897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:36:43.818887 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.818850    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"]
Jul 14 01:36:43.819331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.819206    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf9e2168-2c0f-4f7a-8a69-0c58293ad775" containerName="main"
Jul 14 01:36:43.819331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.819217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf9e2168-2c0f-4f7a-8a69-0c58293ad775" containerName="main"
Jul 14 01:36:43.819331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.819226    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" containerName="maas-api"
Jul 14 01:36:43.819331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.819232    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" containerName="maas-api"
Jul 14 01:36:43.819331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.819288    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3f5fc51-78e7-430a-beb9-6f7d3f2a2466" containerName="maas-api"
Jul 14 01:36:43.819331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.819298    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf9e2168-2c0f-4f7a-8a69-0c58293ad775" containerName="main"
Jul 14 01:36:43.822471 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.822450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:43.824912 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.824886    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:36:43.825024 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.824929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c6eb1c-serving-cert\""
Jul 14 01:36:43.825024 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.824951    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:36:43.825610 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.825587    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:36:43.830995 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.830973    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"]
Jul 14 01:36:43.958027 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.957987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:43.958206 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.958070    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d39d857-e06c-4844-8b67-5104a741fdda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:43.958206 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:43.958136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxnk\" (UniqueName: \"kubernetes.io/projected/3d39d857-e06c-4844-8b67-5104a741fdda-kube-api-access-rpxnk\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.058933 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.058898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.059157 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.058957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d39d857-e06c-4844-8b67-5104a741fdda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.059157 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.059015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxnk\" (UniqueName: \"kubernetes.io/projected/3d39d857-e06c-4844-8b67-5104a741fdda-kube-api-access-rpxnk\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.059157 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:36:44.059036    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-serving-cert: secret "maas-api-e2e-inf-a-c6eb1c-serving-cert" not found
Jul 14 01:36:44.059157 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:36:44.059105    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls podName:3d39d857-e06c-4844-8b67-5104a741fdda nodeName:}" failed. No retries permitted until 2026-07-14 01:36:44.55908783 +0000 UTC m=+2206.446352438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls") pod "maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" (UID: "3d39d857-e06c-4844-8b67-5104a741fdda") : secret "maas-api-e2e-inf-a-c6eb1c-serving-cert" not found
Jul 14 01:36:44.059666 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.059641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d39d857-e06c-4844-8b67-5104a741fdda-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.067222 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.067202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxnk\" (UniqueName: \"kubernetes.io/projected/3d39d857-e06c-4844-8b67-5104a741fdda-kube-api-access-rpxnk\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.562523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.562487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.565015 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.564991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.734660 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.734629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:44.864072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:44.864038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"]
Jul 14 01:36:44.867917 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:36:44.867892    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d39d857_e06c_4844_8b67_5104a741fdda.slice/crio-5a36db42ef363119569559d5dac7b1320fa012a2b51cf57fe1d11a817929b244 WatchSource:0}: Error finding container 5a36db42ef363119569559d5dac7b1320fa012a2b51cf57fe1d11a817929b244: Status 404 returned error can't find the container with id 5a36db42ef363119569559d5dac7b1320fa012a2b51cf57fe1d11a817929b244
Jul 14 01:36:45.722150 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:45.722063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" event={"ID":"3d39d857-e06c-4844-8b67-5104a741fdda","Type":"ContainerStarted","Data":"576045c4917d29f56c8e0b79a796c10b22f9fef92cb68b68d0b944f664b64cc8"}
Jul 14 01:36:45.722150 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:45.722121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" event={"ID":"3d39d857-e06c-4844-8b67-5104a741fdda","Type":"ContainerStarted","Data":"5a36db42ef363119569559d5dac7b1320fa012a2b51cf57fe1d11a817929b244"}
Jul 14 01:36:45.722351 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:45.722244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:45.738214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:45.738166    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" podStartSLOduration=2.161386524 podStartE2EDuration="2.738149965s" totalImagesPullingTime="576.763441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:36:45.736645547 +0000 UTC m=+2207.623910174" watchObservedRunningTime="2026-07-14 01:36:45.738149965 +0000 UTC m=+2207.625414593"
Jul 14 01:36:51.531075 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.531041    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:36:51.733473 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.733444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:36:51.856145 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.856063    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"]
Jul 14 01:36:51.860335 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.860312    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:51.862594 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.862567    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d5004e-serving-cert\""
Jul 14 01:36:51.867152 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.867129    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"]
Jul 14 01:36:51.926592 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.926561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55f16b-0602-4e53-8bea-5bc13ca90754-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:51.926751 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.926706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:51.926797 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:51.926751    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkr9x\" (UniqueName: \"kubernetes.io/projected/1b55f16b-0602-4e53-8bea-5bc13ca90754-kube-api-access-fkr9x\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.028119 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.028088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.028249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.028127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkr9x\" (UniqueName: \"kubernetes.io/projected/1b55f16b-0602-4e53-8bea-5bc13ca90754-kube-api-access-fkr9x\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.028249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.028165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55f16b-0602-4e53-8bea-5bc13ca90754-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.028249 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:36:52.028238    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-serving-cert: secret "maas-api-e2e-inf-b-d5004e-serving-cert" not found
Jul 14 01:36:52.028344 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:36:52.028308    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls podName:1b55f16b-0602-4e53-8bea-5bc13ca90754 nodeName:}" failed. No retries permitted until 2026-07-14 01:36:52.528288776 +0000 UTC m=+2214.415553384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls") pod "maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" (UID: "1b55f16b-0602-4e53-8bea-5bc13ca90754") : secret "maas-api-e2e-inf-b-d5004e-serving-cert" not found
Jul 14 01:36:52.028767 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.028751    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55f16b-0602-4e53-8bea-5bc13ca90754-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.038277 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.038246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkr9x\" (UniqueName: \"kubernetes.io/projected/1b55f16b-0602-4e53-8bea-5bc13ca90754-kube-api-access-fkr9x\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.532623 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.532586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.535186 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.535167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls\") pod \"maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.773574 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.773523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:52.905093 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:52.905069    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"]
Jul 14 01:36:52.907199 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:36:52.907169    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b55f16b_0602_4e53_8bea_5bc13ca90754.slice/crio-cba386446f519843a0fc1770c5260c6ebe8abd7909d17a378bed30b34fc02030 WatchSource:0}: Error finding container cba386446f519843a0fc1770c5260c6ebe8abd7909d17a378bed30b34fc02030: Status 404 returned error can't find the container with id cba386446f519843a0fc1770c5260c6ebe8abd7909d17a378bed30b34fc02030
Jul 14 01:36:53.765528 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:53.765490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" event={"ID":"1b55f16b-0602-4e53-8bea-5bc13ca90754","Type":"ContainerStarted","Data":"28019106c72d41ca6c2935fae852bc6b7cf12f709f9ba1b76b35705ac5a6b926"}
Jul 14 01:36:53.766040 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:53.765551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" event={"ID":"1b55f16b-0602-4e53-8bea-5bc13ca90754","Type":"ContainerStarted","Data":"cba386446f519843a0fc1770c5260c6ebe8abd7909d17a378bed30b34fc02030"}
Jul 14 01:36:53.766040 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:53.765592    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:36:53.781006 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:53.780959    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" podStartSLOduration=2.392041401 podStartE2EDuration="2.780944404s" totalImagesPullingTime="388.903003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:36:53.779970584 +0000 UTC m=+2215.667235213" watchObservedRunningTime="2026-07-14 01:36:53.780944404 +0000 UTC m=+2215.668209031"
Jul 14 01:36:58.762611 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:58.762583    2578 scope.go:117] "RemoveContainer" containerID="faa4eb9337cd50e4a90d0cf6c556f17f86449b6fbd6f4ba6a1a5bb6e324e5768"
Jul 14 01:36:59.776528 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:36:59.776500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:37:06.036339 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:06.036302    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:37:12.471810 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.471778    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"]
Jul 14 01:37:12.474993 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.474975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.477697 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.477657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6eb1c\"/\"test-model-c6eb1c-kserve-self-signed-certs\""
Jul 14 01:37:12.478338 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.478322    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6eb1c\"/\"kube-root-ca.crt\""
Jul 14 01:37:12.478414 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.478391    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6eb1c\"/\"openshift-service-ca.crt\""
Jul 14 01:37:12.478529 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.478509    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6eb1c\"/\"default-dockercfg-r5wt6\""
Jul 14 01:37:12.490992 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.490972    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"]
Jul 14 01:37:12.618117 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.618087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-dshm\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.618263 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.618134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vbz\" (UniqueName: \"kubernetes.io/projected/646f0798-d87c-46e8-9726-e6c8ea92ca93-kube-api-access-n5vbz\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.618263 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.618159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-tmp-dir\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.618263 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.618186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-model-cache\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.618263 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.618209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646f0798-d87c-46e8-9726-e6c8ea92ca93-tls-certs\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.618263 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.618233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-home\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719057 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-home\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-dshm\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vbz\" (UniqueName: \"kubernetes.io/projected/646f0798-d87c-46e8-9726-e6c8ea92ca93-kube-api-access-n5vbz\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-tmp-dir\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-model-cache\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646f0798-d87c-46e8-9726-e6c8ea92ca93-tls-certs\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719468 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-home\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719524 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-tmp-dir\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.719571 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.719545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-model-cache\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.721531 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.721509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-dshm\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.721776 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.721758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646f0798-d87c-46e8-9726-e6c8ea92ca93-tls-certs\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.726346 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.726300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vbz\" (UniqueName: \"kubernetes.io/projected/646f0798-d87c-46e8-9726-e6c8ea92ca93-kube-api-access-n5vbz\") pod \"test-model-c6eb1c-kserve-844b7b84b5-65prz\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") " pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.784570 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.784545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:12.913485 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:12.913460    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"]
Jul 14 01:37:12.915406 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:37:12.915362    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod646f0798_d87c_46e8_9726_e6c8ea92ca93.slice/crio-ff257b85b759964231393824e848fb88e221051ac5c34f32198901f00e988166 WatchSource:0}: Error finding container ff257b85b759964231393824e848fb88e221051ac5c34f32198901f00e988166: Status 404 returned error can't find the container with id ff257b85b759964231393824e848fb88e221051ac5c34f32198901f00e988166
Jul 14 01:37:13.859282 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:13.859243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz" event={"ID":"646f0798-d87c-46e8-9726-e6c8ea92ca93","Type":"ContainerStarted","Data":"0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d"}
Jul 14 01:37:13.859282 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:13.859280    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz" event={"ID":"646f0798-d87c-46e8-9726-e6c8ea92ca93","Type":"ContainerStarted","Data":"ff257b85b759964231393824e848fb88e221051ac5c34f32198901f00e988166"}
Jul 14 01:37:13.875362 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:13.875309    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz" podStartSLOduration=1.875295138 podStartE2EDuration="1.875295138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:37:13.873731572 +0000 UTC m=+2235.760996199" watchObservedRunningTime="2026-07-14 01:37:13.875295138 +0000 UTC m=+2235.762559765"
Jul 14 01:37:16.628210 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:16.628177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:37:22.784831 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:22.784747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:22.785257 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:22.784876    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:22.797325 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:22.797300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:22.915177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:22.915152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:37:28.040342 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:28.040308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:37:30.532010 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.531973    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"]
Jul 14 01:37:30.536784 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.536764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.539153 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.539131    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5004e\"/\"openshift-service-ca.crt\""
Jul 14 01:37:30.540381 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.540357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5004e\"/\"test-model-d5004e-kserve-self-signed-certs\""
Jul 14 01:37:30.540576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.540557    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d5004e\"/\"default-dockercfg-4glnf\""
Jul 14 01:37:30.540663 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.540617    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d5004e\"/\"kube-root-ca.crt\""
Jul 14 01:37:30.542998 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.542973    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"]
Jul 14 01:37:30.676410 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.676381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tls-certs\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.676559 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.676416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-home\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.676559 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.676443    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-model-cache\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.676559 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.676464    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-dshm\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.676767 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.676568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rjg\" (UniqueName: \"kubernetes.io/projected/d53f0af1-1bfb-4402-b6ea-2887d726c28a-kube-api-access-w9rjg\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.676767 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.676649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tmp-dir\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.777885 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.777854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rjg\" (UniqueName: \"kubernetes.io/projected/d53f0af1-1bfb-4402-b6ea-2887d726c28a-kube-api-access-w9rjg\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778021 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.777928    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tmp-dir\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778021 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.777971    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tls-certs\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778021 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.777996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-home\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778192 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.778036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-model-cache\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778192 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.778073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-dshm\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778347 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.778328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tmp-dir\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778421 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.778399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-model-cache\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.778479 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.778411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-home\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.780446 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.780413    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-dshm\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.780578 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.780564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tls-certs\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.785208 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.785168    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rjg\" (UniqueName: \"kubernetes.io/projected/d53f0af1-1bfb-4402-b6ea-2887d726c28a-kube-api-access-w9rjg\") pod \"test-model-d5004e-kserve-56ff86f845-gfz24\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") " pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.848041 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.848004    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:30.984405 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:30.984374    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"]
Jul 14 01:37:30.987903 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:37:30.987840    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd53f0af1_1bfb_4402_b6ea_2887d726c28a.slice/crio-e3f02fc179875120f17a05a5bb3cc5f53779fbaed9833e11c650ab846693c724 WatchSource:0}: Error finding container e3f02fc179875120f17a05a5bb3cc5f53779fbaed9833e11c650ab846693c724: Status 404 returned error can't find the container with id e3f02fc179875120f17a05a5bb3cc5f53779fbaed9833e11c650ab846693c724
Jul 14 01:37:31.953220 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:31.953181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24" event={"ID":"d53f0af1-1bfb-4402-b6ea-2887d726c28a","Type":"ContainerStarted","Data":"cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77"}
Jul 14 01:37:31.953220 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:31.953225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24" event={"ID":"d53f0af1-1bfb-4402-b6ea-2887d726c28a","Type":"ContainerStarted","Data":"e3f02fc179875120f17a05a5bb3cc5f53779fbaed9833e11c650ab846693c724"}
Jul 14 01:37:31.970232 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:31.970186    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24" podStartSLOduration=1.97017255 podStartE2EDuration="1.97017255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:37:31.967587908 +0000 UTC m=+2253.854852537" watchObservedRunningTime="2026-07-14 01:37:31.97017255 +0000 UTC m=+2253.857437179"
Jul 14 01:37:40.848570 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:40.848534    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:40.848960 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:40.848585    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:40.861006 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:40.860980    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:41.009559 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:41.009532    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:37:43.429261 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:43.429225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:37:49.575957 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:49.575918    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"]
Jul 14 01:37:50.460456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:50.460413    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"]
Jul 14 01:37:58.634803 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:37:58.634757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:38:00.950253 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:00.950222    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"]
Jul 14 01:38:00.950652 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:00.950506    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" podUID="3d39d857-e06c-4844-8b67-5104a741fdda" containerName="maas-api" containerID="cri-o://576045c4917d29f56c8e0b79a796c10b22f9fef92cb68b68d0b944f664b64cc8" gracePeriod=30
Jul 14 01:38:01.093403 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.093371    2578 generic.go:356] "Generic (PLEG): container finished" podID="3d39d857-e06c-4844-8b67-5104a741fdda" containerID="576045c4917d29f56c8e0b79a796c10b22f9fef92cb68b68d0b944f664b64cc8" exitCode=0
Jul 14 01:38:01.093544 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.093448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" event={"ID":"3d39d857-e06c-4844-8b67-5104a741fdda","Type":"ContainerDied","Data":"576045c4917d29f56c8e0b79a796c10b22f9fef92cb68b68d0b944f664b64cc8"}
Jul 14 01:38:01.202598 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.202543    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:38:01.347012 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.346984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d39d857-e06c-4844-8b67-5104a741fdda-openshift-service-ca\") pod \"3d39d857-e06c-4844-8b67-5104a741fdda\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") "
Jul 14 01:38:01.347211 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.347047    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls\") pod \"3d39d857-e06c-4844-8b67-5104a741fdda\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") "
Jul 14 01:38:01.347211 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.347074    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpxnk\" (UniqueName: \"kubernetes.io/projected/3d39d857-e06c-4844-8b67-5104a741fdda-kube-api-access-rpxnk\") pod \"3d39d857-e06c-4844-8b67-5104a741fdda\" (UID: \"3d39d857-e06c-4844-8b67-5104a741fdda\") "
Jul 14 01:38:01.347420 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.347391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d39d857-e06c-4844-8b67-5104a741fdda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d39d857-e06c-4844-8b67-5104a741fdda" (UID: "3d39d857-e06c-4844-8b67-5104a741fdda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:38:01.349277 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.349248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d39d857-e06c-4844-8b67-5104a741fdda" (UID: "3d39d857-e06c-4844-8b67-5104a741fdda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:38:01.349373 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.349294    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d39d857-e06c-4844-8b67-5104a741fdda-kube-api-access-rpxnk" (OuterVolumeSpecName: "kube-api-access-rpxnk") pod "3d39d857-e06c-4844-8b67-5104a741fdda" (UID: "3d39d857-e06c-4844-8b67-5104a741fdda"). InnerVolumeSpecName "kube-api-access-rpxnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:38:01.447696 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.447660    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d39d857-e06c-4844-8b67-5104a741fdda-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:01.447856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.447712    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d39d857-e06c-4844-8b67-5104a741fdda-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:01.447856 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:01.447723    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxnk\" (UniqueName: \"kubernetes.io/projected/3d39d857-e06c-4844-8b67-5104a741fdda-kube-api-access-rpxnk\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:02.099534 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:02.099497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4" event={"ID":"3d39d857-e06c-4844-8b67-5104a741fdda","Type":"ContainerDied","Data":"5a36db42ef363119569559d5dac7b1320fa012a2b51cf57fe1d11a817929b244"}
Jul 14 01:38:02.099534 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:02.099529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"
Jul 14 01:38:02.100024 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:02.099574    2578 scope.go:117] "RemoveContainer" containerID="576045c4917d29f56c8e0b79a796c10b22f9fef92cb68b68d0b944f664b64cc8"
Jul 14 01:38:02.119588 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:02.119558    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"]
Jul 14 01:38:02.122955 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:02.122934    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6eb1c-7fb467fcfb-6d2b4"]
Jul 14 01:38:02.691348 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:02.691320    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d39d857-e06c-4844-8b67-5104a741fdda" path="/var/lib/kubelet/pods/3d39d857-e06c-4844-8b67-5104a741fdda/volumes"
Jul 14 01:38:04.597191 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.597147    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz" podUID="646f0798-d87c-46e8-9726-e6c8ea92ca93" containerName="main" containerID="cri-o://0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d" gracePeriod=45
Jul 14 01:38:04.845458 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.845437    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:38:04.976801 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.976772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-model-cache\") pod \"646f0798-d87c-46e8-9726-e6c8ea92ca93\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") "
Jul 14 01:38:04.976948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.976813    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646f0798-d87c-46e8-9726-e6c8ea92ca93-tls-certs\") pod \"646f0798-d87c-46e8-9726-e6c8ea92ca93\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") "
Jul 14 01:38:04.976948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.976859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-dshm\") pod \"646f0798-d87c-46e8-9726-e6c8ea92ca93\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") "
Jul 14 01:38:04.976948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.976884    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5vbz\" (UniqueName: \"kubernetes.io/projected/646f0798-d87c-46e8-9726-e6c8ea92ca93-kube-api-access-n5vbz\") pod \"646f0798-d87c-46e8-9726-e6c8ea92ca93\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") "
Jul 14 01:38:04.976948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.976904    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-home\") pod \"646f0798-d87c-46e8-9726-e6c8ea92ca93\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") "
Jul 14 01:38:04.976948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.976940    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-tmp-dir\") pod \"646f0798-d87c-46e8-9726-e6c8ea92ca93\" (UID: \"646f0798-d87c-46e8-9726-e6c8ea92ca93\") "
Jul 14 01:38:04.977211 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.977036    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-model-cache" (OuterVolumeSpecName: "model-cache") pod "646f0798-d87c-46e8-9726-e6c8ea92ca93" (UID: "646f0798-d87c-46e8-9726-e6c8ea92ca93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:04.977274 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.977215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "646f0798-d87c-46e8-9726-e6c8ea92ca93" (UID: "646f0798-d87c-46e8-9726-e6c8ea92ca93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:04.977274 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.977230    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-home" (OuterVolumeSpecName: "home") pod "646f0798-d87c-46e8-9726-e6c8ea92ca93" (UID: "646f0798-d87c-46e8-9726-e6c8ea92ca93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:04.977370 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.977328    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:04.977370 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.977347    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:04.977370 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.977363    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:04.979299 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.979274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646f0798-d87c-46e8-9726-e6c8ea92ca93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "646f0798-d87c-46e8-9726-e6c8ea92ca93" (UID: "646f0798-d87c-46e8-9726-e6c8ea92ca93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:38:04.979409 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.979309    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-dshm" (OuterVolumeSpecName: "dshm") pod "646f0798-d87c-46e8-9726-e6c8ea92ca93" (UID: "646f0798-d87c-46e8-9726-e6c8ea92ca93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:04.979409 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:04.979325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646f0798-d87c-46e8-9726-e6c8ea92ca93-kube-api-access-n5vbz" (OuterVolumeSpecName: "kube-api-access-n5vbz") pod "646f0798-d87c-46e8-9726-e6c8ea92ca93" (UID: "646f0798-d87c-46e8-9726-e6c8ea92ca93"). InnerVolumeSpecName "kube-api-access-n5vbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:38:05.078889 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.078854    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/646f0798-d87c-46e8-9726-e6c8ea92ca93-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.078889 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.078885    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vbz\" (UniqueName: \"kubernetes.io/projected/646f0798-d87c-46e8-9726-e6c8ea92ca93-kube-api-access-n5vbz\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.078889 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.078896    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/646f0798-d87c-46e8-9726-e6c8ea92ca93-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.120639 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.120602    2578 generic.go:356] "Generic (PLEG): container finished" podID="646f0798-d87c-46e8-9726-e6c8ea92ca93" containerID="0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d" exitCode=0
Jul 14 01:38:05.120833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.120661    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"
Jul 14 01:38:05.120833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.120675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz" event={"ID":"646f0798-d87c-46e8-9726-e6c8ea92ca93","Type":"ContainerDied","Data":"0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d"}
Jul 14 01:38:05.120833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.120744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz" event={"ID":"646f0798-d87c-46e8-9726-e6c8ea92ca93","Type":"ContainerDied","Data":"ff257b85b759964231393824e848fb88e221051ac5c34f32198901f00e988166"}
Jul 14 01:38:05.120833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.120766    2578 scope.go:117] "RemoveContainer" containerID="0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d"
Jul 14 01:38:05.129889 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.129874    2578 scope.go:117] "RemoveContainer" containerID="0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d"
Jul 14 01:38:05.130139 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:38:05.130120    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d\": container with ID starting with 0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d not found: ID does not exist" containerID="0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d"
Jul 14 01:38:05.130204 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.130146    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d"} err="failed to get container status \"0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d\": rpc error: code = NotFound desc = could not find container \"0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d\": container with ID starting with 0ea488e9376c68ca85217f9539b2358b5c73a53a37a5b106ca8803571566288d not found: ID does not exist"
Jul 14 01:38:05.140384 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.140362    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"]
Jul 14 01:38:05.144426 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.144407    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c6eb1c/test-model-c6eb1c-kserve-844b7b84b5-65prz"]
Jul 14 01:38:05.482117 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.482063    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24" podUID="d53f0af1-1bfb-4402-b6ea-2887d726c28a" containerName="main" containerID="cri-o://cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77" gracePeriod=45
Jul 14 01:38:05.722930 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.722905    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:38:05.785456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9rjg\" (UniqueName: \"kubernetes.io/projected/d53f0af1-1bfb-4402-b6ea-2887d726c28a-kube-api-access-w9rjg\") pod \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") "
Jul 14 01:38:05.785456 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-dshm\") pod \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") "
Jul 14 01:38:05.785649 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-home\") pod \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") "
Jul 14 01:38:05.785649 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-model-cache\") pod \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") "
Jul 14 01:38:05.785649 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tls-certs\") pod \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") "
Jul 14 01:38:05.785649 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tmp-dir\") pod \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\" (UID: \"d53f0af1-1bfb-4402-b6ea-2887d726c28a\") "
Jul 14 01:38:05.785882 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785811    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-home" (OuterVolumeSpecName: "home") pod "d53f0af1-1bfb-4402-b6ea-2887d726c28a" (UID: "d53f0af1-1bfb-4402-b6ea-2887d726c28a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:05.785935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785840    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-model-cache" (OuterVolumeSpecName: "model-cache") pod "d53f0af1-1bfb-4402-b6ea-2887d726c28a" (UID: "d53f0af1-1bfb-4402-b6ea-2887d726c28a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:05.785935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785925    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d53f0af1-1bfb-4402-b6ea-2887d726c28a" (UID: "d53f0af1-1bfb-4402-b6ea-2887d726c28a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:05.786034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.785979    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.786034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.786000    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.787893 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.787870    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-dshm" (OuterVolumeSpecName: "dshm") pod "d53f0af1-1bfb-4402-b6ea-2887d726c28a" (UID: "d53f0af1-1bfb-4402-b6ea-2887d726c28a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:38:05.788003 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.787868    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d53f0af1-1bfb-4402-b6ea-2887d726c28a" (UID: "d53f0af1-1bfb-4402-b6ea-2887d726c28a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:38:05.788003 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.787911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53f0af1-1bfb-4402-b6ea-2887d726c28a-kube-api-access-w9rjg" (OuterVolumeSpecName: "kube-api-access-w9rjg") pod "d53f0af1-1bfb-4402-b6ea-2887d726c28a" (UID: "d53f0af1-1bfb-4402-b6ea-2887d726c28a"). InnerVolumeSpecName "kube-api-access-w9rjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:38:05.886859 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.886831    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.887000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.886874    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.887000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.886884    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rjg\" (UniqueName: \"kubernetes.io/projected/d53f0af1-1bfb-4402-b6ea-2887d726c28a-kube-api-access-w9rjg\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:05.887000 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:05.886895    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d53f0af1-1bfb-4402-b6ea-2887d726c28a-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:06.127795 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.127710    2578 generic.go:356] "Generic (PLEG): container finished" podID="d53f0af1-1bfb-4402-b6ea-2887d726c28a" containerID="cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77" exitCode=0
Jul 14 01:38:06.127795 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.127755    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"
Jul 14 01:38:06.128177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.127755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24" event={"ID":"d53f0af1-1bfb-4402-b6ea-2887d726c28a","Type":"ContainerDied","Data":"cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77"}
Jul 14 01:38:06.128177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.127880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24" event={"ID":"d53f0af1-1bfb-4402-b6ea-2887d726c28a","Type":"ContainerDied","Data":"e3f02fc179875120f17a05a5bb3cc5f53779fbaed9833e11c650ab846693c724"}
Jul 14 01:38:06.128177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.127901    2578 scope.go:117] "RemoveContainer" containerID="cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77"
Jul 14 01:38:06.139545 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.139522    2578 scope.go:117] "RemoveContainer" containerID="cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77"
Jul 14 01:38:06.139850 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:38:06.139824    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77\": container with ID starting with cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77 not found: ID does not exist" containerID="cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77"
Jul 14 01:38:06.139953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.139854    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77"} err="failed to get container status \"cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77\": rpc error: code = NotFound desc = could not find container \"cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77\": container with ID starting with cc490101e85965d15b4193d1e179c8377e15a4ea0452445e2bb00b43abe02d77 not found: ID does not exist"
Jul 14 01:38:06.148672 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.148649    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"]
Jul 14 01:38:06.152312 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.152289    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d5004e/test-model-d5004e-kserve-56ff86f845-gfz24"]
Jul 14 01:38:06.691820 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.691784    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646f0798-d87c-46e8-9726-e6c8ea92ca93" path="/var/lib/kubelet/pods/646f0798-d87c-46e8-9726-e6c8ea92ca93/volumes"
Jul 14 01:38:06.692184 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:06.692171    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53f0af1-1bfb-4402-b6ea-2887d726c28a" path="/var/lib/kubelet/pods/d53f0af1-1bfb-4402-b6ea-2887d726c28a/volumes"
Jul 14 01:38:09.732045 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:09.732008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:38:27.062551 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.062510    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"]
Jul 14 01:38:27.063085 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.062770    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" podUID="1b55f16b-0602-4e53-8bea-5bc13ca90754" containerName="maas-api" containerID="cri-o://28019106c72d41ca6c2935fae852bc6b7cf12f709f9ba1b76b35705ac5a6b926" gracePeriod=30
Jul 14 01:38:27.237841 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.237811    2578 generic.go:356] "Generic (PLEG): container finished" podID="1b55f16b-0602-4e53-8bea-5bc13ca90754" containerID="28019106c72d41ca6c2935fae852bc6b7cf12f709f9ba1b76b35705ac5a6b926" exitCode=0
Jul 14 01:38:27.237993 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.237872    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" event={"ID":"1b55f16b-0602-4e53-8bea-5bc13ca90754","Type":"ContainerDied","Data":"28019106c72d41ca6c2935fae852bc6b7cf12f709f9ba1b76b35705ac5a6b926"}
Jul 14 01:38:27.309372 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.309349    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:38:27.468204 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.468176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkr9x\" (UniqueName: \"kubernetes.io/projected/1b55f16b-0602-4e53-8bea-5bc13ca90754-kube-api-access-fkr9x\") pod \"1b55f16b-0602-4e53-8bea-5bc13ca90754\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") "
Jul 14 01:38:27.468362 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.468252    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls\") pod \"1b55f16b-0602-4e53-8bea-5bc13ca90754\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") "
Jul 14 01:38:27.468362 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.468278    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55f16b-0602-4e53-8bea-5bc13ca90754-openshift-service-ca\") pod \"1b55f16b-0602-4e53-8bea-5bc13ca90754\" (UID: \"1b55f16b-0602-4e53-8bea-5bc13ca90754\") "
Jul 14 01:38:27.468708 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.468649    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b55f16b-0602-4e53-8bea-5bc13ca90754-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1b55f16b-0602-4e53-8bea-5bc13ca90754" (UID: "1b55f16b-0602-4e53-8bea-5bc13ca90754"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:38:27.470358 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.470338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b55f16b-0602-4e53-8bea-5bc13ca90754-kube-api-access-fkr9x" (OuterVolumeSpecName: "kube-api-access-fkr9x") pod "1b55f16b-0602-4e53-8bea-5bc13ca90754" (UID: "1b55f16b-0602-4e53-8bea-5bc13ca90754"). InnerVolumeSpecName "kube-api-access-fkr9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:38:27.470471 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.470337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1b55f16b-0602-4e53-8bea-5bc13ca90754" (UID: "1b55f16b-0602-4e53-8bea-5bc13ca90754"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:38:27.569093 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.569065    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b55f16b-0602-4e53-8bea-5bc13ca90754-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:27.569093 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.569091    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkr9x\" (UniqueName: \"kubernetes.io/projected/1b55f16b-0602-4e53-8bea-5bc13ca90754-kube-api-access-fkr9x\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:27.569258 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:27.569100    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b55f16b-0602-4e53-8bea-5bc13ca90754-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:38:28.243631 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:28.243592    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5" event={"ID":"1b55f16b-0602-4e53-8bea-5bc13ca90754","Type":"ContainerDied","Data":"cba386446f519843a0fc1770c5260c6ebe8abd7909d17a378bed30b34fc02030"}
Jul 14 01:38:28.244052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:28.243650    2578 scope.go:117] "RemoveContainer" containerID="28019106c72d41ca6c2935fae852bc6b7cf12f709f9ba1b76b35705ac5a6b926"
Jul 14 01:38:28.244052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:28.243608    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"
Jul 14 01:38:28.262562 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:28.262541    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"]
Jul 14 01:38:28.266948 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:28.266923    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d5004e-7b8dc4f5b5-92dg5"]
Jul 14 01:38:28.690908 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:28.690876    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b55f16b-0602-4e53-8bea-5bc13ca90754" path="/var/lib/kubelet/pods/1b55f16b-0602-4e53-8bea-5bc13ca90754/volumes"
Jul 14 01:38:29.930062 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:29.930029    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:38:46.376101 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376068    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"]
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376409    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53f0af1-1bfb-4402-b6ea-2887d726c28a" containerName="main"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376419    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53f0af1-1bfb-4402-b6ea-2887d726c28a" containerName="main"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376431    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b55f16b-0602-4e53-8bea-5bc13ca90754" containerName="maas-api"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376436    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b55f16b-0602-4e53-8bea-5bc13ca90754" containerName="maas-api"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376443    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646f0798-d87c-46e8-9726-e6c8ea92ca93" containerName="main"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376449    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="646f0798-d87c-46e8-9726-e6c8ea92ca93" containerName="main"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376459    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d39d857-e06c-4844-8b67-5104a741fdda" containerName="maas-api"
Jul 14 01:38:46.376464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376464    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d39d857-e06c-4844-8b67-5104a741fdda" containerName="maas-api"
Jul 14 01:38:46.376758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376520    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="646f0798-d87c-46e8-9726-e6c8ea92ca93" containerName="main"
Jul 14 01:38:46.376758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376531    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d39d857-e06c-4844-8b67-5104a741fdda" containerName="maas-api"
Jul 14 01:38:46.376758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376537    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53f0af1-1bfb-4402-b6ea-2887d726c28a" containerName="main"
Jul 14 01:38:46.376758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.376542    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b55f16b-0602-4e53-8bea-5bc13ca90754" containerName="maas-api"
Jul 14 01:38:46.379643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.379626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.381922 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.381899    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ef50e2-serving-cert\""
Jul 14 01:38:46.381922 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.381910    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:38:46.382696 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.382659    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:38:46.382801 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.382660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:38:46.390318 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.390298    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"]
Jul 14 01:38:46.431260 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.431227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.431375 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.431279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2bn\" (UniqueName: \"kubernetes.io/projected/4315d0d1-7685-4bca-b5a9-df465186639e-kube-api-access-pk2bn\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.431428 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.431409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4315d0d1-7685-4bca-b5a9-df465186639e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.531927 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.531889    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4315d0d1-7685-4bca-b5a9-df465186639e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.531927 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.531934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.532082 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.531984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2bn\" (UniqueName: \"kubernetes.io/projected/4315d0d1-7685-4bca-b5a9-df465186639e-kube-api-access-pk2bn\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.532127 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:38:46.532103    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-serving-cert: secret "maas-api-e2e-api-a-ef50e2-serving-cert" not found
Jul 14 01:38:46.532224 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:38:46.532171    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls podName:4315d0d1-7685-4bca-b5a9-df465186639e nodeName:}" failed. No retries permitted until 2026-07-14 01:38:47.032153758 +0000 UTC m=+2328.919418367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls") pod "maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" (UID: "4315d0d1-7685-4bca-b5a9-df465186639e") : secret "maas-api-e2e-api-a-ef50e2-serving-cert" not found
Jul 14 01:38:46.532585 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.532566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4315d0d1-7685-4bca-b5a9-df465186639e-openshift-service-ca\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:46.553157 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:46.553128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2bn\" (UniqueName: \"kubernetes.io/projected/4315d0d1-7685-4bca-b5a9-df465186639e-kube-api-access-pk2bn\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:47.035785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:47.035748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:47.038146 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:47.038123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls\") pod \"maas-api-e2e-api-a-ef50e2-58ffbff66-789cd\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:47.291530 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:47.291443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:47.434236 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:47.434209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"]
Jul 14 01:38:47.437167 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:38:47.437125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4315d0d1_7685_4bca_b5a9_df465186639e.slice/crio-cca67e07d5e7934d1b10fc97c0a97fe1abab36415fd7bd657118939f7acb3592 WatchSource:0}: Error finding container cca67e07d5e7934d1b10fc97c0a97fe1abab36415fd7bd657118939f7acb3592: Status 404 returned error can't find the container with id cca67e07d5e7934d1b10fc97c0a97fe1abab36415fd7bd657118939f7acb3592
Jul 14 01:38:47.438750 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:47.438717    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:38:48.340343 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:48.340253    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" event={"ID":"4315d0d1-7685-4bca-b5a9-df465186639e","Type":"ContainerStarted","Data":"b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52"}
Jul 14 01:38:48.340343 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:48.340314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" event={"ID":"4315d0d1-7685-4bca-b5a9-df465186639e","Type":"ContainerStarted","Data":"cca67e07d5e7934d1b10fc97c0a97fe1abab36415fd7bd657118939f7acb3592"}
Jul 14 01:38:48.340343 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:48.340339    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:48.359419 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:48.359366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" podStartSLOduration=1.881534656 podStartE2EDuration="2.359350789s" totalImagesPullingTime="477.816133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:38:48.357502857 +0000 UTC m=+2330.244767486" watchObservedRunningTime="2026-07-14 01:38:48.359350789 +0000 UTC m=+2330.246615416"
Jul 14 01:38:51.132999 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:51.132917    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:38:54.351107 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.351072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:38:54.559718 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.559673    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"]
Jul 14 01:38:54.563284 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.563262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.565974 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.565951    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-99bb7c-serving-cert\""
Jul 14 01:38:54.584020 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.583997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"]
Jul 14 01:38:54.705963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.705930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.705963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.705961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7lq\" (UniqueName: \"kubernetes.io/projected/40e94c3e-a710-43b8-bebb-d202e1561450-kube-api-access-hp7lq\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.706187 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.705997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e94c3e-a710-43b8-bebb-d202e1561450-openshift-service-ca\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.806939 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.806904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.806939 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.806937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7lq\" (UniqueName: \"kubernetes.io/projected/40e94c3e-a710-43b8-bebb-d202e1561450-kube-api-access-hp7lq\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.807164 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.806977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e94c3e-a710-43b8-bebb-d202e1561450-openshift-service-ca\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.807164 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:38:54.807072    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-serving-cert: secret "maas-api-e2e-api-b-99bb7c-serving-cert" not found
Jul 14 01:38:54.807164 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:38:54.807149    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls podName:40e94c3e-a710-43b8-bebb-d202e1561450 nodeName:}" failed. No retries permitted until 2026-07-14 01:38:55.307131686 +0000 UTC m=+2337.194396291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls") pod "maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" (UID: "40e94c3e-a710-43b8-bebb-d202e1561450") : secret "maas-api-e2e-api-b-99bb7c-serving-cert" not found
Jul 14 01:38:54.807705 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.807668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e94c3e-a710-43b8-bebb-d202e1561450-openshift-service-ca\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:54.815079 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:54.815049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7lq\" (UniqueName: \"kubernetes.io/projected/40e94c3e-a710-43b8-bebb-d202e1561450-kube-api-access-hp7lq\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:55.312231 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:55.312189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:55.314768 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:55.314749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls\") pod \"maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:55.473377 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:55.473335    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:55.601817 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:55.601794    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"]
Jul 14 01:38:55.603629 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:38:55.603564    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e94c3e_a710_43b8_bebb_d202e1561450.slice/crio-0293c5353a77129225c15ee004c10b0b1d585e1528c536d6a7cadcb60a0dad7f WatchSource:0}: Error finding container 0293c5353a77129225c15ee004c10b0b1d585e1528c536d6a7cadcb60a0dad7f: Status 404 returned error can't find the container with id 0293c5353a77129225c15ee004c10b0b1d585e1528c536d6a7cadcb60a0dad7f
Jul 14 01:38:56.382675 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:56.382594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" event={"ID":"40e94c3e-a710-43b8-bebb-d202e1561450","Type":"ContainerStarted","Data":"0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd"}
Jul 14 01:38:56.382675 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:56.382655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" event={"ID":"40e94c3e-a710-43b8-bebb-d202e1561450","Type":"ContainerStarted","Data":"0293c5353a77129225c15ee004c10b0b1d585e1528c536d6a7cadcb60a0dad7f"}
Jul 14 01:38:56.382675 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:56.382677    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:38:56.399190 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:38:56.399147    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" podStartSLOduration=2.04603582 podStartE2EDuration="2.399133618s" totalImagesPullingTime="353.097798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:38:56.397100629 +0000 UTC m=+2338.284365258" watchObservedRunningTime="2026-07-14 01:38:56.399133618 +0000 UTC m=+2338.286398223"
Jul 14 01:39:02.393184 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:02.393156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:39:07.103025 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.102988    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"]
Jul 14 01:39:07.106439 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.106422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:07.108565 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.108544    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zn7l8\""
Jul 14 01:39:07.112033 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.112009    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"]
Jul 14 01:39:07.216123 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.216082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f45d67-6a19-40a0-9ed2-533187f733cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd\" (UID: \"29f45d67-6a19-40a0-9ed2-533187f733cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:07.317075 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.317035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f45d67-6a19-40a0-9ed2-533187f733cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd\" (UID: \"29f45d67-6a19-40a0-9ed2-533187f733cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:07.317668 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.317649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f45d67-6a19-40a0-9ed2-533187f733cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd\" (UID: \"29f45d67-6a19-40a0-9ed2-533187f733cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:07.418293 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.418267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:07.551260 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.551234    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"]
Jul 14 01:39:07.553340 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:39:07.553315    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f45d67_6a19_40a0_9ed2_533187f733cc.slice/crio-01eb7d0baf0fb7aebe15da9593147b23f7106ddc4888883b04cc69450ed1ddfd WatchSource:0}: Error finding container 01eb7d0baf0fb7aebe15da9593147b23f7106ddc4888883b04cc69450ed1ddfd: Status 404 returned error can't find the container with id 01eb7d0baf0fb7aebe15da9593147b23f7106ddc4888883b04cc69450ed1ddfd
Jul 14 01:39:07.628462 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:07.628430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:39:08.440082 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:08.440049    2578 generic.go:356] "Generic (PLEG): container finished" podID="29f45d67-6a19-40a0-9ed2-533187f733cc" containerID="6089ada111552baaf123b4d10f144398e0c772b09a0ec3ae41cd8dfde5b5895c" exitCode=0
Jul 14 01:39:08.440463 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:08.440101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd" event={"ID":"29f45d67-6a19-40a0-9ed2-533187f733cc","Type":"ContainerDied","Data":"6089ada111552baaf123b4d10f144398e0c772b09a0ec3ae41cd8dfde5b5895c"}
Jul 14 01:39:08.440463 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:08.440142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd" event={"ID":"29f45d67-6a19-40a0-9ed2-533187f733cc","Type":"ContainerStarted","Data":"01eb7d0baf0fb7aebe15da9593147b23f7106ddc4888883b04cc69450ed1ddfd"}
Jul 14 01:39:09.576742 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:09.576717    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:09.738890 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:09.738792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f45d67-6a19-40a0-9ed2-533187f733cc-maas-api-service-ca\") pod \"29f45d67-6a19-40a0-9ed2-533187f733cc\" (UID: \"29f45d67-6a19-40a0-9ed2-533187f733cc\") "
Jul 14 01:39:09.739171 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:09.739146    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f45d67-6a19-40a0-9ed2-533187f733cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29f45d67-6a19-40a0-9ed2-533187f733cc" (UID: "29f45d67-6a19-40a0-9ed2-533187f733cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:39:09.839934 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:09.839892    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f45d67-6a19-40a0-9ed2-533187f733cc-maas-api-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:10.450549 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:10.450521    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd"
Jul 14 01:39:10.450737 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:10.450518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd" event={"ID":"29f45d67-6a19-40a0-9ed2-533187f733cc","Type":"ContainerDied","Data":"01eb7d0baf0fb7aebe15da9593147b23f7106ddc4888883b04cc69450ed1ddfd"}
Jul 14 01:39:10.450737 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:10.450651    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01eb7d0baf0fb7aebe15da9593147b23f7106ddc4888883b04cc69450ed1ddfd"
Jul 14 01:39:17.976563 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:17.976530    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"]
Jul 14 01:39:17.977050 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:17.976793    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" podUID="4315d0d1-7685-4bca-b5a9-df465186639e" containerName="maas-api" containerID="cri-o://b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52" gracePeriod=30
Jul 14 01:39:18.223101 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.223077    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:39:18.319160 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.319076    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls\") pod \"4315d0d1-7685-4bca-b5a9-df465186639e\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") "
Jul 14 01:39:18.319160 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.319118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4315d0d1-7685-4bca-b5a9-df465186639e-openshift-service-ca\") pod \"4315d0d1-7685-4bca-b5a9-df465186639e\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") "
Jul 14 01:39:18.319160 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.319151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk2bn\" (UniqueName: \"kubernetes.io/projected/4315d0d1-7685-4bca-b5a9-df465186639e-kube-api-access-pk2bn\") pod \"4315d0d1-7685-4bca-b5a9-df465186639e\" (UID: \"4315d0d1-7685-4bca-b5a9-df465186639e\") "
Jul 14 01:39:18.319509 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.319484    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4315d0d1-7685-4bca-b5a9-df465186639e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4315d0d1-7685-4bca-b5a9-df465186639e" (UID: "4315d0d1-7685-4bca-b5a9-df465186639e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:39:18.321354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.321330    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4315d0d1-7685-4bca-b5a9-df465186639e-kube-api-access-pk2bn" (OuterVolumeSpecName: "kube-api-access-pk2bn") pod "4315d0d1-7685-4bca-b5a9-df465186639e" (UID: "4315d0d1-7685-4bca-b5a9-df465186639e"). InnerVolumeSpecName "kube-api-access-pk2bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:39:18.321462 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.321434    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4315d0d1-7685-4bca-b5a9-df465186639e" (UID: "4315d0d1-7685-4bca-b5a9-df465186639e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:39:18.420158 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.420118    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4315d0d1-7685-4bca-b5a9-df465186639e-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:18.420158 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.420152    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4315d0d1-7685-4bca-b5a9-df465186639e-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:18.420158 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.420163    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2bn\" (UniqueName: \"kubernetes.io/projected/4315d0d1-7685-4bca-b5a9-df465186639e-kube-api-access-pk2bn\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:18.490002 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.489969    2578 generic.go:356] "Generic (PLEG): container finished" podID="4315d0d1-7685-4bca-b5a9-df465186639e" containerID="b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52" exitCode=0
Jul 14 01:39:18.490185 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.490015    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"
Jul 14 01:39:18.490185 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.490056    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" event={"ID":"4315d0d1-7685-4bca-b5a9-df465186639e","Type":"ContainerDied","Data":"b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52"}
Jul 14 01:39:18.490185 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.490105    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd" event={"ID":"4315d0d1-7685-4bca-b5a9-df465186639e","Type":"ContainerDied","Data":"cca67e07d5e7934d1b10fc97c0a97fe1abab36415fd7bd657118939f7acb3592"}
Jul 14 01:39:18.490185 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.490126    2578 scope.go:117] "RemoveContainer" containerID="b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52"
Jul 14 01:39:18.499214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.499198    2578 scope.go:117] "RemoveContainer" containerID="b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52"
Jul 14 01:39:18.499463 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:39:18.499444    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52\": container with ID starting with b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52 not found: ID does not exist" containerID="b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52"
Jul 14 01:39:18.499524 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.499474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52"} err="failed to get container status \"b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52\": rpc error: code = NotFound desc = could not find container \"b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52\": container with ID starting with b422a2dde783caa05acf9dff7e5160743d4258052afd8aea847b887a40ab0b52 not found: ID does not exist"
Jul 14 01:39:18.510173 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.510153    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"]
Jul 14 01:39:18.513626 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.513606    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ef50e2-58ffbff66-789cd"]
Jul 14 01:39:18.691158 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:18.691124    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4315d0d1-7685-4bca-b5a9-df465186639e" path="/var/lib/kubelet/pods/4315d0d1-7685-4bca-b5a9-df465186639e/volumes"
Jul 14 01:39:23.130942 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:23.130908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:39:33.240462 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240428    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"]
Jul 14 01:39:33.240963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240839    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29f45d67-6a19-40a0-9ed2-533187f733cc" containerName="revoke-keys"
Jul 14 01:39:33.240963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240852    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="29f45d67-6a19-40a0-9ed2-533187f733cc" containerName="revoke-keys"
Jul 14 01:39:33.240963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240864    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4315d0d1-7685-4bca-b5a9-df465186639e" containerName="maas-api"
Jul 14 01:39:33.240963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240870    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4315d0d1-7685-4bca-b5a9-df465186639e" containerName="maas-api"
Jul 14 01:39:33.240963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240931    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4315d0d1-7685-4bca-b5a9-df465186639e" containerName="maas-api"
Jul 14 01:39:33.240963 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.240943    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="29f45d67-6a19-40a0-9ed2-533187f733cc" containerName="revoke-keys"
Jul 14 01:39:33.243877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.243860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:33.247181 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.247162    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zn7l8\""
Jul 14 01:39:33.261433 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.261409    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"]
Jul 14 01:39:33.346251 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.346212    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978dde70-998d-477e-a63e-c72cf2f47f94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5\" (UID: \"978dde70-998d-477e-a63e-c72cf2f47f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:33.446935 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.446904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978dde70-998d-477e-a63e-c72cf2f47f94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5\" (UID: \"978dde70-998d-477e-a63e-c72cf2f47f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:33.447515 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.447496    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978dde70-998d-477e-a63e-c72cf2f47f94-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5\" (UID: \"978dde70-998d-477e-a63e-c72cf2f47f94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:33.553551 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.553483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:33.681156 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:33.681131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"]
Jul 14 01:39:33.683671 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:39:33.683641    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978dde70_998d_477e_a63e_c72cf2f47f94.slice/crio-312a907e5c28d9cb82cf71a8cbef0cbebb18fa9c1622e09238f01c0d89d39bfa WatchSource:0}: Error finding container 312a907e5c28d9cb82cf71a8cbef0cbebb18fa9c1622e09238f01c0d89d39bfa: Status 404 returned error can't find the container with id 312a907e5c28d9cb82cf71a8cbef0cbebb18fa9c1622e09238f01c0d89d39bfa
Jul 14 01:39:34.566331 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:34.566299    2578 generic.go:356] "Generic (PLEG): container finished" podID="978dde70-998d-477e-a63e-c72cf2f47f94" containerID="055cf0e00e618926e69f46fbc9c019f70a512959b3e7fe888f2e818e01f35bb1" exitCode=0
Jul 14 01:39:34.566668 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:34.566359    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5" event={"ID":"978dde70-998d-477e-a63e-c72cf2f47f94","Type":"ContainerDied","Data":"055cf0e00e618926e69f46fbc9c019f70a512959b3e7fe888f2e818e01f35bb1"}
Jul 14 01:39:34.566668 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:34.566405    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5" event={"ID":"978dde70-998d-477e-a63e-c72cf2f47f94","Type":"ContainerStarted","Data":"312a907e5c28d9cb82cf71a8cbef0cbebb18fa9c1622e09238f01c0d89d39bfa"}
Jul 14 01:39:35.698458 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:35.698434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:35.766574 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:35.766546    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978dde70-998d-477e-a63e-c72cf2f47f94-maas-api-service-ca\") pod \"978dde70-998d-477e-a63e-c72cf2f47f94\" (UID: \"978dde70-998d-477e-a63e-c72cf2f47f94\") "
Jul 14 01:39:35.767326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:35.767253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978dde70-998d-477e-a63e-c72cf2f47f94-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "978dde70-998d-477e-a63e-c72cf2f47f94" (UID: "978dde70-998d-477e-a63e-c72cf2f47f94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:39:35.873947 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:35.873878    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/978dde70-998d-477e-a63e-c72cf2f47f94-maas-api-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:36.577241 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:36.577213    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5"
Jul 14 01:39:36.577413 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:36.577211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5" event={"ID":"978dde70-998d-477e-a63e-c72cf2f47f94","Type":"ContainerDied","Data":"312a907e5c28d9cb82cf71a8cbef0cbebb18fa9c1622e09238f01c0d89d39bfa"}
Jul 14 01:39:36.577413 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:36.577363    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312a907e5c28d9cb82cf71a8cbef0cbebb18fa9c1622e09238f01c0d89d39bfa"
Jul 14 01:39:43.309251 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.309220    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"]
Jul 14 01:39:43.309669 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.309413    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" podUID="40e94c3e-a710-43b8-bebb-d202e1561450" containerName="maas-api" containerID="cri-o://0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd" gracePeriod=30
Jul 14 01:39:43.560855 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.560791    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:39:43.610560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.610528    2578 generic.go:356] "Generic (PLEG): container finished" podID="40e94c3e-a710-43b8-bebb-d202e1561450" containerID="0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd" exitCode=0
Jul 14 01:39:43.610706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.610571    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"
Jul 14 01:39:43.610706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.610612    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" event={"ID":"40e94c3e-a710-43b8-bebb-d202e1561450","Type":"ContainerDied","Data":"0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd"}
Jul 14 01:39:43.610706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.610695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw" event={"ID":"40e94c3e-a710-43b8-bebb-d202e1561450","Type":"ContainerDied","Data":"0293c5353a77129225c15ee004c10b0b1d585e1528c536d6a7cadcb60a0dad7f"}
Jul 14 01:39:43.610819 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.610729    2578 scope.go:117] "RemoveContainer" containerID="0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd"
Jul 14 01:39:43.618774 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.618749    2578 scope.go:117] "RemoveContainer" containerID="0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd"
Jul 14 01:39:43.619008 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:39:43.618992    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd\": container with ID starting with 0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd not found: ID does not exist" containerID="0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd"
Jul 14 01:39:43.619063 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.619015    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd"} err="failed to get container status \"0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd\": rpc error: code = NotFound desc = could not find container \"0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd\": container with ID starting with 0b57b32078fef584b6242b1b8b48a6d16a2d381c5821ca4479f5c1d4afb0d3dd not found: ID does not exist"
Jul 14 01:39:43.641280 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.641258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls\") pod \"40e94c3e-a710-43b8-bebb-d202e1561450\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") "
Jul 14 01:39:43.641361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.641325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e94c3e-a710-43b8-bebb-d202e1561450-openshift-service-ca\") pod \"40e94c3e-a710-43b8-bebb-d202e1561450\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") "
Jul 14 01:39:43.641361 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.641356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp7lq\" (UniqueName: \"kubernetes.io/projected/40e94c3e-a710-43b8-bebb-d202e1561450-kube-api-access-hp7lq\") pod \"40e94c3e-a710-43b8-bebb-d202e1561450\" (UID: \"40e94c3e-a710-43b8-bebb-d202e1561450\") "
Jul 14 01:39:43.641647 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.641626    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e94c3e-a710-43b8-bebb-d202e1561450-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "40e94c3e-a710-43b8-bebb-d202e1561450" (UID: "40e94c3e-a710-43b8-bebb-d202e1561450"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:39:43.643352 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.643328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "40e94c3e-a710-43b8-bebb-d202e1561450" (UID: "40e94c3e-a710-43b8-bebb-d202e1561450"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:39:43.643452 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.643401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e94c3e-a710-43b8-bebb-d202e1561450-kube-api-access-hp7lq" (OuterVolumeSpecName: "kube-api-access-hp7lq") pod "40e94c3e-a710-43b8-bebb-d202e1561450" (UID: "40e94c3e-a710-43b8-bebb-d202e1561450"). InnerVolumeSpecName "kube-api-access-hp7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:39:43.742100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.742072    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40e94c3e-a710-43b8-bebb-d202e1561450-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:43.742100 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.742096    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e94c3e-a710-43b8-bebb-d202e1561450-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:43.742277 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.742106    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp7lq\" (UniqueName: \"kubernetes.io/projected/40e94c3e-a710-43b8-bebb-d202e1561450-kube-api-access-hp7lq\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:39:43.931116 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.931089    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"]
Jul 14 01:39:43.933130 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:43.933110    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-99bb7c-6745bfcf56-rt4sw"]
Jul 14 01:39:44.691356 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:44.691323    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e94c3e-a710-43b8-bebb-d202e1561450" path="/var/lib/kubelet/pods/40e94c3e-a710-43b8-bebb-d202e1561450/volumes"
Jul 14 01:39:46.631368 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:46.631337    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:39:58.734121 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:58.734085    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:39:58.743124 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:39:58.743101    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:40:02.827916 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.827883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"]
Jul 14 01:40:02.828438 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.828418    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e94c3e-a710-43b8-bebb-d202e1561450" containerName="maas-api"
Jul 14 01:40:02.828513 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.828440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e94c3e-a710-43b8-bebb-d202e1561450" containerName="maas-api"
Jul 14 01:40:02.828513 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.828456    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="978dde70-998d-477e-a63e-c72cf2f47f94" containerName="revoke-keys"
Jul 14 01:40:02.828513 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.828463    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="978dde70-998d-477e-a63e-c72cf2f47f94" containerName="revoke-keys"
Jul 14 01:40:02.828664 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.828569    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e94c3e-a710-43b8-bebb-d202e1561450" containerName="maas-api"
Jul 14 01:40:02.828664 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.828587    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="978dde70-998d-477e-a63e-c72cf2f47f94" containerName="revoke-keys"
Jul 14 01:40:02.831543 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.831522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:02.833765 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.833745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:40:02.834480 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.834457    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:40:02.834568 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.834477    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nvgxn\""
Jul 14 01:40:02.834568 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.834458    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c6543c-serving-cert\""
Jul 14 01:40:02.842466 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:02.842445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"]
Jul 14 01:40:03.008968 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.008932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569740af-a93b-4f26-b3b2-7944a77b9af7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.009133 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.009011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.009133 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.009050    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zk82\" (UniqueName: \"kubernetes.io/projected/569740af-a93b-4f26-b3b2-7944a77b9af7-kube-api-access-2zk82\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.110072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.109997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zk82\" (UniqueName: \"kubernetes.io/projected/569740af-a93b-4f26-b3b2-7944a77b9af7-kube-api-access-2zk82\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.110072 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.110048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569740af-a93b-4f26-b3b2-7944a77b9af7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.110251 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.110105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.110251 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:40:03.110209    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-serving-cert: secret "maas-api-e2e-shared-a-c6543c-serving-cert" not found
Jul 14 01:40:03.110322 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:40:03.110269    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls podName:569740af-a93b-4f26-b3b2-7944a77b9af7 nodeName:}" failed. No retries permitted until 2026-07-14 01:40:03.610254164 +0000 UTC m=+2405.497518770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls") pod "maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" (UID: "569740af-a93b-4f26-b3b2-7944a77b9af7") : secret "maas-api-e2e-shared-a-c6543c-serving-cert" not found
Jul 14 01:40:03.110663 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.110643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569740af-a93b-4f26-b3b2-7944a77b9af7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.118089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.118068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zk82\" (UniqueName: \"kubernetes.io/projected/569740af-a93b-4f26-b3b2-7944a77b9af7-kube-api-access-2zk82\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.614907 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.614871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.617333 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.617315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls\") pod \"maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.742643 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.742606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:03.871674 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:03.871604    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"]
Jul 14 01:40:03.875274 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:40:03.875243    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569740af_a93b_4f26_b3b2_7944a77b9af7.slice/crio-0b307e9ecd34af31123e32cf7bbab54622bb3477342c953b5b3c05f28f3f2846 WatchSource:0}: Error finding container 0b307e9ecd34af31123e32cf7bbab54622bb3477342c953b5b3c05f28f3f2846: Status 404 returned error can't find the container with id 0b307e9ecd34af31123e32cf7bbab54622bb3477342c953b5b3c05f28f3f2846
Jul 14 01:40:04.714080 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:04.714031    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" event={"ID":"569740af-a93b-4f26-b3b2-7944a77b9af7","Type":"ContainerStarted","Data":"44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498"}
Jul 14 01:40:04.714321 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:04.714100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" event={"ID":"569740af-a93b-4f26-b3b2-7944a77b9af7","Type":"ContainerStarted","Data":"0b307e9ecd34af31123e32cf7bbab54622bb3477342c953b5b3c05f28f3f2846"}
Jul 14 01:40:04.714321 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:04.714147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:04.756120 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:04.755851    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" podStartSLOduration=2.347277267 podStartE2EDuration="2.755831238s" totalImagesPullingTime="408.553971ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:40:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:40:04.754877232 +0000 UTC m=+2406.642141860" watchObservedRunningTime="2026-07-14 01:40:04.755831238 +0000 UTC m=+2406.643095868"
Jul 14 01:40:08.136318 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:08.136280    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:40:10.724199 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:10.724168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:40:10.980938 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:10.980861    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"]
Jul 14 01:40:10.984571 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:10.984550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:10.987143 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:10.987123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a21fee-serving-cert\""
Jul 14 01:40:10.992215 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:10.992193    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"]
Jul 14 01:40:11.083747 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.083716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31e0ee-e8e9-408f-b55d-50160fe8a436-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.083921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.083833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4wx\" (UniqueName: \"kubernetes.io/projected/5a31e0ee-e8e9-408f-b55d-50160fe8a436-kube-api-access-pc4wx\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.083921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.083878    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31e0ee-e8e9-408f-b55d-50160fe8a436-maas-api-tls\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.185123 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.185078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4wx\" (UniqueName: \"kubernetes.io/projected/5a31e0ee-e8e9-408f-b55d-50160fe8a436-kube-api-access-pc4wx\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.185289 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.185136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31e0ee-e8e9-408f-b55d-50160fe8a436-maas-api-tls\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.185289 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.185163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31e0ee-e8e9-408f-b55d-50160fe8a436-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.185860 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.185841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31e0ee-e8e9-408f-b55d-50160fe8a436-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.187798 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.187779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31e0ee-e8e9-408f-b55d-50160fe8a436-maas-api-tls\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.192993 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.192972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4wx\" (UniqueName: \"kubernetes.io/projected/5a31e0ee-e8e9-408f-b55d-50160fe8a436-kube-api-access-pc4wx\") pod \"maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.296386 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.296307    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:11.428973 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.428949    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"]
Jul 14 01:40:11.432040 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:40:11.432014    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a31e0ee_e8e9_408f_b55d_50160fe8a436.slice/crio-9091315113ca2f0faab6b8720d5c7e6dd712e36d6d54f6ec5f548e420cde2399 WatchSource:0}: Error finding container 9091315113ca2f0faab6b8720d5c7e6dd712e36d6d54f6ec5f548e420cde2399: Status 404 returned error can't find the container with id 9091315113ca2f0faab6b8720d5c7e6dd712e36d6d54f6ec5f548e420cde2399
Jul 14 01:40:11.748658 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:11.748622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx" event={"ID":"5a31e0ee-e8e9-408f-b55d-50160fe8a436","Type":"ContainerStarted","Data":"9091315113ca2f0faab6b8720d5c7e6dd712e36d6d54f6ec5f548e420cde2399"}
Jul 14 01:40:12.754058 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:12.754020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx" event={"ID":"5a31e0ee-e8e9-408f-b55d-50160fe8a436","Type":"ContainerStarted","Data":"fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255"}
Jul 14 01:40:12.754473 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:12.754126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:12.779811 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:12.779762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx" podStartSLOduration=2.331300238 podStartE2EDuration="2.779749611s" totalImagesPullingTime="448.449373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:40:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:40:12.779337931 +0000 UTC m=+2414.666602560" watchObservedRunningTime="2026-07-14 01:40:12.779749611 +0000 UTC m=+2414.667014238"
Jul 14 01:40:18.764230 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:18.764152    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:40:23.335220 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:23.335182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:40:27.452724 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.452664    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"]
Jul 14 01:40:27.456576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.456558    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.458930 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.458908    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c6543c\"/\"default-dockercfg-ccd9s\""
Jul 14 01:40:27.459670 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.459652    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c6543c\"/\"kube-root-ca.crt\""
Jul 14 01:40:27.459785 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.459766    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c6543c\"/\"auth-test-model-c6543c-kserve-self-signed-certs\""
Jul 14 01:40:27.459853 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.459833    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c6543c\"/\"openshift-service-ca.crt\""
Jul 14 01:40:27.465161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.465127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"]
Jul 14 01:40:27.537498 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.537470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tmp-dir\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.537498 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.537500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tls-certs\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.537709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.537531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-model-cache\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.537709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.537549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-dshm\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.537709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.537627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxqf\" (UniqueName: \"kubernetes.io/projected/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-kube-api-access-zjxqf\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.537709 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.537657    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-home\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638286 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxqf\" (UniqueName: \"kubernetes.io/projected/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-kube-api-access-zjxqf\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638423 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-home\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638423 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tmp-dir\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638494 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tls-certs\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638529 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638490    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-model-cache\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-dshm\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638752 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tmp-dir\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638799 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-home\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.638952 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.638926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-model-cache\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.640822 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.640801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-dshm\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.641145 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.641130    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tls-certs\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.645814 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.645796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxqf\" (UniqueName: \"kubernetes.io/projected/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-kube-api-access-zjxqf\") pod \"auth-test-model-c6543c-kserve-68ffffbbc5-wp68f\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") " pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.768154 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.768067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:27.900373 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:27.900348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"]
Jul 14 01:40:27.902162 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:40:27.902138    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod585f7cfe_a2a7_4e19_b3c2_d15a1ae1a72b.slice/crio-6da36dd986d481d5991882328dc42e33d5819ed25dd670d5189b40f771f17bb5 WatchSource:0}: Error finding container 6da36dd986d481d5991882328dc42e33d5819ed25dd670d5189b40f771f17bb5: Status 404 returned error can't find the container with id 6da36dd986d481d5991882328dc42e33d5819ed25dd670d5189b40f771f17bb5
Jul 14 01:40:28.830191 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:28.830162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f" event={"ID":"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b","Type":"ContainerStarted","Data":"659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4"}
Jul 14 01:40:28.830191 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:28.830198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f" event={"ID":"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b","Type":"ContainerStarted","Data":"6da36dd986d481d5991882328dc42e33d5819ed25dd670d5189b40f771f17bb5"}
Jul 14 01:40:28.846481 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:28.846435    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f" podStartSLOduration=1.846419632 podStartE2EDuration="1.846419632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:40:28.845207246 +0000 UTC m=+2430.732471875" watchObservedRunningTime="2026-07-14 01:40:28.846419632 +0000 UTC m=+2430.733684259"
Jul 14 01:40:31.226584 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:31.226549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:40:37.769073 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:37.769041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:37.769564 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:37.769087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:37.781627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:37.781606    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:40:37.890766 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:40:37.890740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:41:14.830573 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.830540    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"]
Jul 14 01:41:14.834454 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.834433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:14.836878 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.836677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"openshift-service-ca.crt\""
Jul 14 01:41:14.837824 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.837781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"auth-test-model-a21fee-kserve-self-signed-certs\""
Jul 14 01:41:14.837949 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.837849    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"default-dockercfg-c69hq\""
Jul 14 01:41:14.837949 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.837785    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"kube-root-ca.crt\""
Jul 14 01:41:14.841561 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.841540    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"]
Jul 14 01:41:14.932202 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.932173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-home\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:14.932344 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.932222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-model-cache\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:14.932344 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.932271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jf6j\" (UniqueName: \"kubernetes.io/projected/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-kube-api-access-6jf6j\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:14.932344 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.932316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tmp-dir\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:14.932344 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.932340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tls-certs\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:14.932520 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:14.932367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-dshm\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.033670 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.033639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-home\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.033819 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.033727    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-model-cache\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.033819 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.033773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jf6j\" (UniqueName: \"kubernetes.io/projected/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-kube-api-access-6jf6j\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.033819 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.033809    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tmp-dir\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.033996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.033839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tls-certs\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.033996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.033879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-dshm\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.034097 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.034019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-home\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.034169 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.034102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-model-cache\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.034233 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.034197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tmp-dir\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.036324 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.036306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-dshm\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.036524 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.036509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tls-certs\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.041356 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.041332    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jf6j\" (UniqueName: \"kubernetes.io/projected/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-kube-api-access-6jf6j\") pod \"auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") " pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.145675 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.145595    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:15.276506 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:15.276482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"]
Jul 14 01:41:15.278729 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:41:15.278696    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd42052c_3c5c_4caa_b0a3_2f8973d8544c.slice/crio-5534cc30726a0f4f64735dbc8d685074cf5de8f6870e32ea7327a1d3f924812b WatchSource:0}: Error finding container 5534cc30726a0f4f64735dbc8d685074cf5de8f6870e32ea7327a1d3f924812b: Status 404 returned error can't find the container with id 5534cc30726a0f4f64735dbc8d685074cf5de8f6870e32ea7327a1d3f924812b
Jul 14 01:41:16.062158 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:16.062124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n" event={"ID":"fd42052c-3c5c-4caa-b0a3-2f8973d8544c","Type":"ContainerStarted","Data":"dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02"}
Jul 14 01:41:16.062524 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:16.062169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n" event={"ID":"fd42052c-3c5c-4caa-b0a3-2f8973d8544c","Type":"ContainerStarted","Data":"5534cc30726a0f4f64735dbc8d685074cf5de8f6870e32ea7327a1d3f924812b"}
Jul 14 01:41:16.079372 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:16.079315    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n" podStartSLOduration=2.079300506 podStartE2EDuration="2.079300506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:41:16.078112971 +0000 UTC m=+2477.965377599" watchObservedRunningTime="2026-07-14 01:41:16.079300506 +0000 UTC m=+2477.966565130"
Jul 14 01:41:18.531649 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:18.531613    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:41:25.146726 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:25.146661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:25.147117 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:25.146749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:25.159244 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:25.159212    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:26.119202 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:26.119175    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:41:32.736695 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:32.736660    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:41:51.740720 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:41:51.740618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:42:08.334555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:42:08.334513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:42:25.728924 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:42:25.728886    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:42:42.129092 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:42:42.129056    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:42:59.838338 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:42:59.838297    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:43:11.291786 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:11.291750    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"]
Jul 14 01:43:12.208405 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:12.208371    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"]
Jul 14 01:43:15.925510 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:15.925474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:43:18.625001 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.624966    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"]
Jul 14 01:43:18.628613 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.628589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.630953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.630931    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c6543c\"/\"sub-test-model-c6543c-kserve-self-signed-certs\""
Jul 14 01:43:18.638706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.638666    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"]
Jul 14 01:43:18.743301 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.743227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtcp\" (UniqueName: \"kubernetes.io/projected/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-kube-api-access-tqtcp\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.743301 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.743259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-home\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.743301 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.743280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-dshm\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.743604 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.743321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-model-cache\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.743604 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.743455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tmp-dir\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.743604 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.743501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tls-certs\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844434 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844403    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tmp-dir\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844631 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844440    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tls-certs\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844631 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtcp\" (UniqueName: \"kubernetes.io/projected/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-kube-api-access-tqtcp\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844631 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-home\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844631 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844561    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-dshm\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844866 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-model-cache\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844862    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-home\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.844941 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.844915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tmp-dir\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.845051 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.845033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-model-cache\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.846886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.846870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-dshm\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.847172 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.847154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tls-certs\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.852121 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.852100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtcp\" (UniqueName: \"kubernetes.io/projected/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-kube-api-access-tqtcp\") pod \"sub-test-model-c6543c-kserve-868b87b747-tfsh9\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") " pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:18.938155 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:18.938118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:19.073745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:19.073721    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"]
Jul 14 01:43:19.076031 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:43:19.076007    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2e08baa_2150_46e1_9ffe_6d720d1ddc06.slice/crio-825a63f9f81daef5ad0dd83a6453e2d8beae34f8d8a1e73bcb462a1d96bbab0e WatchSource:0}: Error finding container 825a63f9f81daef5ad0dd83a6453e2d8beae34f8d8a1e73bcb462a1d96bbab0e: Status 404 returned error can't find the container with id 825a63f9f81daef5ad0dd83a6453e2d8beae34f8d8a1e73bcb462a1d96bbab0e
Jul 14 01:43:19.649715 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:19.649665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9" event={"ID":"c2e08baa-2150-46e1-9ffe-6d720d1ddc06","Type":"ContainerStarted","Data":"f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5"}
Jul 14 01:43:19.650096 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:19.649725    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9" event={"ID":"c2e08baa-2150-46e1-9ffe-6d720d1ddc06","Type":"ContainerStarted","Data":"825a63f9f81daef5ad0dd83a6453e2d8beae34f8d8a1e73bcb462a1d96bbab0e"}
Jul 14 01:43:19.667321 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:19.667279    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9" podStartSLOduration=1.6672656080000001 podStartE2EDuration="1.667265608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:43:19.665303968 +0000 UTC m=+2601.552568597" watchObservedRunningTime="2026-07-14 01:43:19.667265608 +0000 UTC m=+2601.554530214"
Jul 14 01:43:26.312594 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.312545    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f" podUID="585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" containerName="main" containerID="cri-o://659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4" gracePeriod=45
Jul 14 01:43:26.562733 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.562664    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:43:26.615048 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615017    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tls-certs\") pod \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") "
Jul 14 01:43:26.615202 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tmp-dir\") pod \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") "
Jul 14 01:43:26.615202 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjxqf\" (UniqueName: \"kubernetes.io/projected/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-kube-api-access-zjxqf\") pod \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") "
Jul 14 01:43:26.615202 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-model-cache\") pod \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") "
Jul 14 01:43:26.615338 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615202    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-dshm\") pod \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") "
Jul 14 01:43:26.615338 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-home\") pod \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\" (UID: \"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b\") "
Jul 14 01:43:26.615440 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615371    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" (UID: "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:26.615497 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-model-cache" (OuterVolumeSpecName: "model-cache") pod "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" (UID: "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:26.615561 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-home" (OuterVolumeSpecName: "home") pod "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" (UID: "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:26.615626 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615610    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:26.615669 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.615631    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:26.617449 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.617421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" (UID: "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:43:26.617449 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.617437    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-kube-api-access-zjxqf" (OuterVolumeSpecName: "kube-api-access-zjxqf") pod "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" (UID: "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b"). InnerVolumeSpecName "kube-api-access-zjxqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:43:26.617591 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.617485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-dshm" (OuterVolumeSpecName: "dshm") pod "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" (UID: "585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:26.696836 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.696802    2578 generic.go:356] "Generic (PLEG): container finished" podID="585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" containerID="659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4" exitCode=0
Jul 14 01:43:26.696977 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.696930    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"
Jul 14 01:43:26.698044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.698018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f" event={"ID":"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b","Type":"ContainerDied","Data":"659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4"}
Jul 14 01:43:26.698132 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.698060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f" event={"ID":"585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b","Type":"ContainerDied","Data":"6da36dd986d481d5991882328dc42e33d5819ed25dd670d5189b40f771f17bb5"}
Jul 14 01:43:26.698132 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.698084    2578 scope.go:117] "RemoveContainer" containerID="659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4"
Jul 14 01:43:26.708141 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.708125    2578 scope.go:117] "RemoveContainer" containerID="659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4"
Jul 14 01:43:26.708384 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:43:26.708367    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4\": container with ID starting with 659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4 not found: ID does not exist" containerID="659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4"
Jul 14 01:43:26.708450 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.708393    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4"} err="failed to get container status \"659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4\": rpc error: code = NotFound desc = could not find container \"659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4\": container with ID starting with 659f060e3ba62dfccab6537e6c685947f5825bced4b6becc11c1e86dfb3db8a4 not found: ID does not exist"
Jul 14 01:43:26.716489 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.716467    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:26.716588 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.716495    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxqf\" (UniqueName: \"kubernetes.io/projected/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-kube-api-access-zjxqf\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:26.716588 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.716510    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:26.716588 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.716579    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:26.718597 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.718577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"]
Jul 14 01:43:26.721996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:26.721978    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/auth-test-model-c6543c-kserve-68ffffbbc5-wp68f"]
Jul 14 01:43:27.230804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.230764    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n" podUID="fd42052c-3c5c-4caa-b0a3-2f8973d8544c" containerName="main" containerID="cri-o://dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02" gracePeriod=45
Jul 14 01:43:27.483405 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.483345    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:43:27.624727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624670    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-model-cache\") pod \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") "
Jul 14 01:43:27.624895 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tls-certs\") pod \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") "
Jul 14 01:43:27.624895 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624830    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jf6j\" (UniqueName: \"kubernetes.io/projected/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-kube-api-access-6jf6j\") pod \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") "
Jul 14 01:43:27.624895 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-dshm\") pod \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") "
Jul 14 01:43:27.625034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624943    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tmp-dir\") pod \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") "
Jul 14 01:43:27.625034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd42052c-3c5c-4caa-b0a3-2f8973d8544c" (UID: "fd42052c-3c5c-4caa-b0a3-2f8973d8544c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:27.625034 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.624978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-home\") pod \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\" (UID: \"fd42052c-3c5c-4caa-b0a3-2f8973d8544c\") "
Jul 14 01:43:27.625213 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.625161    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd42052c-3c5c-4caa-b0a3-2f8973d8544c" (UID: "fd42052c-3c5c-4caa-b0a3-2f8973d8544c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:27.625276 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.625233    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-home" (OuterVolumeSpecName: "home") pod "fd42052c-3c5c-4caa-b0a3-2f8973d8544c" (UID: "fd42052c-3c5c-4caa-b0a3-2f8973d8544c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:27.625316 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.625299    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:27.625316 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.625312    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:27.625384 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.625320    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:27.627200 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.627178    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-dshm" (OuterVolumeSpecName: "dshm") pod "fd42052c-3c5c-4caa-b0a3-2f8973d8544c" (UID: "fd42052c-3c5c-4caa-b0a3-2f8973d8544c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:43:27.627514 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.627495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd42052c-3c5c-4caa-b0a3-2f8973d8544c" (UID: "fd42052c-3c5c-4caa-b0a3-2f8973d8544c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:43:27.627569 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.627505    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-kube-api-access-6jf6j" (OuterVolumeSpecName: "kube-api-access-6jf6j") pod "fd42052c-3c5c-4caa-b0a3-2f8973d8544c" (UID: "fd42052c-3c5c-4caa-b0a3-2f8973d8544c"). InnerVolumeSpecName "kube-api-access-6jf6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:43:27.703729 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.703675    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd42052c-3c5c-4caa-b0a3-2f8973d8544c" containerID="dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02" exitCode=0
Jul 14 01:43:27.703884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.703737    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"
Jul 14 01:43:27.703884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.703731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n" event={"ID":"fd42052c-3c5c-4caa-b0a3-2f8973d8544c","Type":"ContainerDied","Data":"dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02"}
Jul 14 01:43:27.703884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.703851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n" event={"ID":"fd42052c-3c5c-4caa-b0a3-2f8973d8544c","Type":"ContainerDied","Data":"5534cc30726a0f4f64735dbc8d685074cf5de8f6870e32ea7327a1d3f924812b"}
Jul 14 01:43:27.703884 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.703870    2578 scope.go:117] "RemoveContainer" containerID="dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02"
Jul 14 01:43:27.712975 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.712957    2578 scope.go:117] "RemoveContainer" containerID="dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02"
Jul 14 01:43:27.713230 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:43:27.713214    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02\": container with ID starting with dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02 not found: ID does not exist" containerID="dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02"
Jul 14 01:43:27.713273 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.713237    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02"} err="failed to get container status \"dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02\": rpc error: code = NotFound desc = could not find container \"dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02\": container with ID starting with dc66d139f5eb682238f2e809caa647e657bdf0394950e7ffe57ecfa1d41e0e02 not found: ID does not exist"
Jul 14 01:43:27.724169 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.724147    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"]
Jul 14 01:43:27.725897 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.725874    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:27.725897 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.725902    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jf6j\" (UniqueName: \"kubernetes.io/projected/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-kube-api-access-6jf6j\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:27.726048 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.725916    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd42052c-3c5c-4caa-b0a3-2f8973d8544c-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:43:27.727696 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:27.727667    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/auth-test-model-a21fee-kserve-7f9c85dd59-q2x5n"]
Jul 14 01:43:28.691517 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:28.691485    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" path="/var/lib/kubelet/pods/585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b/volumes"
Jul 14 01:43:28.691910 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:28.691895    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd42052c-3c5c-4caa-b0a3-2f8973d8544c" path="/var/lib/kubelet/pods/fd42052c-3c5c-4caa-b0a3-2f8973d8544c/volumes"
Jul 14 01:43:28.939025 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:28.938985    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:28.939222 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:28.939038    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:28.951479 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:28.951413    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:29.725589 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:29.725556    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:43:31.936702 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:31.936647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:43:50.133104 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133063    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"]
Jul 14 01:43:50.133776 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133756    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd42052c-3c5c-4caa-b0a3-2f8973d8544c" containerName="main"
Jul 14 01:43:50.133776 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133778    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd42052c-3c5c-4caa-b0a3-2f8973d8544c" containerName="main"
Jul 14 01:43:50.133947 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133851    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" containerName="main"
Jul 14 01:43:50.133947 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133861    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" containerName="main"
Jul 14 01:43:50.134047 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd42052c-3c5c-4caa-b0a3-2f8973d8544c" containerName="main"
Jul 14 01:43:50.134047 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.133982    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="585f7cfe-a2a7-4e19-b3c2-d15a1ae1a72b" containerName="main"
Jul 14 01:43:50.137147 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.137126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.140246 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.140224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"default-dockercfg-c69hq\""
Jul 14 01:43:50.140371 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.140335    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"openshift-service-ca.crt\""
Jul 14 01:43:50.140445 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.140397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"sub-test-model-a21fee-kserve-self-signed-certs\""
Jul 14 01:43:50.140509 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.140483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"kube-root-ca.crt\""
Jul 14 01:43:50.145824 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.145799    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"]
Jul 14 01:43:50.234428 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.234396    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkmm\" (UniqueName: \"kubernetes.io/projected/c8e61847-08b7-4b3e-b342-8207d699fba2-kube-api-access-qhkmm\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.234590 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.234438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-home\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.234590 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.234466    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e61847-08b7-4b3e-b342-8207d699fba2-tls-certs\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.234590 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.234552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-dshm\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.234788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.234588    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-tmp-dir\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.234788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.234639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-model-cache\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.335529 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-model-cache\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.335706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkmm\" (UniqueName: \"kubernetes.io/projected/c8e61847-08b7-4b3e-b342-8207d699fba2-kube-api-access-qhkmm\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.335706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-home\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.335706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e61847-08b7-4b3e-b342-8207d699fba2-tls-certs\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.335706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-dshm\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.335706 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-tmp-dir\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.336011 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-model-cache\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.336011 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.335999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-home\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.336160 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.336042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-tmp-dir\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.338005 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.337985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-dshm\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.338411 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.338390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e61847-08b7-4b3e-b342-8207d699fba2-tls-certs\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.343953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.343933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkmm\" (UniqueName: \"kubernetes.io/projected/c8e61847-08b7-4b3e-b342-8207d699fba2-kube-api-access-qhkmm\") pod \"sub-test-model-a21fee-kserve-67d794c95-dngtl\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") " pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.447695 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.447667    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:43:50.583088 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.583062    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"]
Jul 14 01:43:50.584053 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:43:50.584028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8e61847_08b7_4b3e_b342_8207d699fba2.slice/crio-8f947eff379767c711c01f480c7d6541269a59fe350892893c7cb452bbae9380 WatchSource:0}: Error finding container 8f947eff379767c711c01f480c7d6541269a59fe350892893c7cb452bbae9380: Status 404 returned error can't find the container with id 8f947eff379767c711c01f480c7d6541269a59fe350892893c7cb452bbae9380
Jul 14 01:43:50.812162 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.812081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl" event={"ID":"c8e61847-08b7-4b3e-b342-8207d699fba2","Type":"ContainerStarted","Data":"40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76"}
Jul 14 01:43:50.812162 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.812118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl" event={"ID":"c8e61847-08b7-4b3e-b342-8207d699fba2","Type":"ContainerStarted","Data":"8f947eff379767c711c01f480c7d6541269a59fe350892893c7cb452bbae9380"}
Jul 14 01:43:50.828484 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:50.828434    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl" podStartSLOduration=0.828420147 podStartE2EDuration="828.420147ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:43:50.826752263 +0000 UTC m=+2632.714016894" watchObservedRunningTime="2026-07-14 01:43:50.828420147 +0000 UTC m=+2632.715684776"
Jul 14 01:43:54.629403 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:43:54.629360    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:44:00.448122 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:00.448086    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:44:00.448566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:00.448365    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:44:00.461453 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:00.461428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:44:00.875329 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:00.875248    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:44:15.728909 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:15.728878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:44:35.532155 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:35.532117    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:44:41.688809 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:41.688775    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"]
Jul 14 01:44:42.517528 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:42.517495    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"]
Jul 14 01:44:54.333530 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.333460    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"]
Jul 14 01:44:54.337224 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.337207    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.339402 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.339381    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c6543c\"/\"rate-test-model-c6543c-kserve-self-signed-certs\""
Jul 14 01:44:54.346571 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.346547    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"]
Jul 14 01:44:54.396830 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.396799    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-dshm\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.396830 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.396834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-model-cache\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.396996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.396854    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-home\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.396996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.396871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-tmp-dir\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.396996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.396968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5qp\" (UniqueName: \"kubernetes.io/projected/7920cac1-cd14-4033-9320-05d208d6e470-kube-api-access-vp5qp\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.397101 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.397052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7920cac1-cd14-4033-9320-05d208d6e470-tls-certs\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.498528 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.498492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5qp\" (UniqueName: \"kubernetes.io/projected/7920cac1-cd14-4033-9320-05d208d6e470-kube-api-access-vp5qp\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.498745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.498556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7920cac1-cd14-4033-9320-05d208d6e470-tls-certs\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.498745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.498599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-dshm\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.498745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.498620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-model-cache\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.498745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.498639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-home\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.498745 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.498661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-tmp-dir\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.499094 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.499073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-model-cache\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.499153 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.499127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-home\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.499153 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.499146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-tmp-dir\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.501166 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.501137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-dshm\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.501275 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.501258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7920cac1-cd14-4033-9320-05d208d6e470-tls-certs\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.505657 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.505639    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5qp\" (UniqueName: \"kubernetes.io/projected/7920cac1-cd14-4033-9320-05d208d6e470-kube-api-access-vp5qp\") pod \"rate-test-model-c6543c-kserve-5bb7c6955-b8vcm\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") " pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.647177 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.647095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:44:54.781286 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:54.781260    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"]
Jul 14 01:44:54.783329 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:44:54.783300    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7920cac1_cd14_4033_9320_05d208d6e470.slice/crio-7744575d72b4a2bd5af6e3cd2fa23d034b787d9ff917337432d003152ebf58f5 WatchSource:0}: Error finding container 7744575d72b4a2bd5af6e3cd2fa23d034b787d9ff917337432d003152ebf58f5: Status 404 returned error can't find the container with id 7744575d72b4a2bd5af6e3cd2fa23d034b787d9ff917337432d003152ebf58f5
Jul 14 01:44:55.130558 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:55.130516    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm" event={"ID":"7920cac1-cd14-4033-9320-05d208d6e470","Type":"ContainerStarted","Data":"985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d"}
Jul 14 01:44:55.130749 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:55.130568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm" event={"ID":"7920cac1-cd14-4033-9320-05d208d6e470","Type":"ContainerStarted","Data":"7744575d72b4a2bd5af6e3cd2fa23d034b787d9ff917337432d003152ebf58f5"}
Jul 14 01:44:55.147421 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:55.147365    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm" podStartSLOduration=1.147349981 podStartE2EDuration="1.147349981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:44:55.145028238 +0000 UTC m=+2697.032292891" watchObservedRunningTime="2026-07-14 01:44:55.147349981 +0000 UTC m=+2697.034614657"
Jul 14 01:44:55.936087 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:55.936049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:44:56.709703 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:56.709659    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9" podUID="c2e08baa-2150-46e1-9ffe-6d720d1ddc06" containerName="main" containerID="cri-o://f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5" gracePeriod=45
Jul 14 01:44:56.965241 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:56.965178    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:44:57.023239 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023210    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-dshm\") pod \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") "
Jul 14 01:44:57.023397 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023248    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tls-certs\") pod \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") "
Jul 14 01:44:57.023397 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023305    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-home\") pod \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") "
Jul 14 01:44:57.023520 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqtcp\" (UniqueName: \"kubernetes.io/projected/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-kube-api-access-tqtcp\") pod \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") "
Jul 14 01:44:57.023520 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-model-cache\") pod \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") "
Jul 14 01:44:57.023629 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tmp-dir\") pod \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\" (UID: \"c2e08baa-2150-46e1-9ffe-6d720d1ddc06\") "
Jul 14 01:44:57.023629 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023585    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-home" (OuterVolumeSpecName: "home") pod "c2e08baa-2150-46e1-9ffe-6d720d1ddc06" (UID: "c2e08baa-2150-46e1-9ffe-6d720d1ddc06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.023827 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023802    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2e08baa-2150-46e1-9ffe-6d720d1ddc06" (UID: "c2e08baa-2150-46e1-9ffe-6d720d1ddc06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.023887 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2e08baa-2150-46e1-9ffe-6d720d1ddc06" (UID: "c2e08baa-2150-46e1-9ffe-6d720d1ddc06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.023962 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023948    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.023998 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023968    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.023998 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.023981    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.025743 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.025719    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-kube-api-access-tqtcp" (OuterVolumeSpecName: "kube-api-access-tqtcp") pod "c2e08baa-2150-46e1-9ffe-6d720d1ddc06" (UID: "c2e08baa-2150-46e1-9ffe-6d720d1ddc06"). InnerVolumeSpecName "kube-api-access-tqtcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:44:57.025969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.025941    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2e08baa-2150-46e1-9ffe-6d720d1ddc06" (UID: "c2e08baa-2150-46e1-9ffe-6d720d1ddc06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:44:57.025969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.025954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-dshm" (OuterVolumeSpecName: "dshm") pod "c2e08baa-2150-46e1-9ffe-6d720d1ddc06" (UID: "c2e08baa-2150-46e1-9ffe-6d720d1ddc06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.124541 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.124517    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqtcp\" (UniqueName: \"kubernetes.io/projected/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-kube-api-access-tqtcp\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.124541 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.124537    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.124541 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.124546    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2e08baa-2150-46e1-9ffe-6d720d1ddc06-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.141470 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.141441    2578 generic.go:356] "Generic (PLEG): container finished" podID="c2e08baa-2150-46e1-9ffe-6d720d1ddc06" containerID="f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5" exitCode=0
Jul 14 01:44:57.141564 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.141490    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"
Jul 14 01:44:57.141564 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.141510    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9" event={"ID":"c2e08baa-2150-46e1-9ffe-6d720d1ddc06","Type":"ContainerDied","Data":"f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5"}
Jul 14 01:44:57.141636 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.141547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9" event={"ID":"c2e08baa-2150-46e1-9ffe-6d720d1ddc06","Type":"ContainerDied","Data":"825a63f9f81daef5ad0dd83a6453e2d8beae34f8d8a1e73bcb462a1d96bbab0e"}
Jul 14 01:44:57.141636 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.141588    2578 scope.go:117] "RemoveContainer" containerID="f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5"
Jul 14 01:44:57.150905 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.150891    2578 scope.go:117] "RemoveContainer" containerID="f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5"
Jul 14 01:44:57.151153 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:44:57.151137    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5\": container with ID starting with f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5 not found: ID does not exist" containerID="f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5"
Jul 14 01:44:57.151192 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.151162    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5"} err="failed to get container status \"f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5\": rpc error: code = NotFound desc = could not find container \"f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5\": container with ID starting with f0d58e762099711ea15dedbd0630d2f654409800473034f1307f7eb6fa1f34b5 not found: ID does not exist"
Jul 14 01:44:57.161143 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.161116    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"]
Jul 14 01:44:57.164847 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.164828    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/sub-test-model-c6543c-kserve-868b87b747-tfsh9"]
Jul 14 01:44:57.537837 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.537799    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl" podUID="c8e61847-08b7-4b3e-b342-8207d699fba2" containerName="main" containerID="cri-o://40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76" gracePeriod=45
Jul 14 01:44:57.797573 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.797516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:44:57.931414 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e61847-08b7-4b3e-b342-8207d699fba2-tls-certs\") pod \"c8e61847-08b7-4b3e-b342-8207d699fba2\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") "
Jul 14 01:44:57.931414 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931414    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-tmp-dir\") pod \"c8e61847-08b7-4b3e-b342-8207d699fba2\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") "
Jul 14 01:44:57.931662 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-home\") pod \"c8e61847-08b7-4b3e-b342-8207d699fba2\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") "
Jul 14 01:44:57.931662 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931471    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-dshm\") pod \"c8e61847-08b7-4b3e-b342-8207d699fba2\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") "
Jul 14 01:44:57.931662 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931487    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-model-cache\") pod \"c8e61847-08b7-4b3e-b342-8207d699fba2\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") "
Jul 14 01:44:57.931662 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhkmm\" (UniqueName: \"kubernetes.io/projected/c8e61847-08b7-4b3e-b342-8207d699fba2-kube-api-access-qhkmm\") pod \"c8e61847-08b7-4b3e-b342-8207d699fba2\" (UID: \"c8e61847-08b7-4b3e-b342-8207d699fba2\") "
Jul 14 01:44:57.931906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8e61847-08b7-4b3e-b342-8207d699fba2" (UID: "c8e61847-08b7-4b3e-b342-8207d699fba2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.931906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931740    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-home" (OuterVolumeSpecName: "home") pod "c8e61847-08b7-4b3e-b342-8207d699fba2" (UID: "c8e61847-08b7-4b3e-b342-8207d699fba2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.931906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8e61847-08b7-4b3e-b342-8207d699fba2" (UID: "c8e61847-08b7-4b3e-b342-8207d699fba2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.931906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931883    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.931906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.931900    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:57.933804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.933777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-dshm" (OuterVolumeSpecName: "dshm") pod "c8e61847-08b7-4b3e-b342-8207d699fba2" (UID: "c8e61847-08b7-4b3e-b342-8207d699fba2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:44:57.933804 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.933785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e61847-08b7-4b3e-b342-8207d699fba2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8e61847-08b7-4b3e-b342-8207d699fba2" (UID: "c8e61847-08b7-4b3e-b342-8207d699fba2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:44:57.933953 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:57.933816    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e61847-08b7-4b3e-b342-8207d699fba2-kube-api-access-qhkmm" (OuterVolumeSpecName: "kube-api-access-qhkmm") pod "c8e61847-08b7-4b3e-b342-8207d699fba2" (UID: "c8e61847-08b7-4b3e-b342-8207d699fba2"). InnerVolumeSpecName "kube-api-access-qhkmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:44:58.032726 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.032698    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhkmm\" (UniqueName: \"kubernetes.io/projected/c8e61847-08b7-4b3e-b342-8207d699fba2-kube-api-access-qhkmm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:58.032726 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.032723    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e61847-08b7-4b3e-b342-8207d699fba2-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:58.033098 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.032735    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:58.033098 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.032744    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8e61847-08b7-4b3e-b342-8207d699fba2-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:44:58.152074 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.151992    2578 generic.go:356] "Generic (PLEG): container finished" podID="c8e61847-08b7-4b3e-b342-8207d699fba2" containerID="40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76" exitCode=0
Jul 14 01:44:58.152360 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.152338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl" event={"ID":"c8e61847-08b7-4b3e-b342-8207d699fba2","Type":"ContainerDied","Data":"40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76"}
Jul 14 01:44:58.152482 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.152467    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl" event={"ID":"c8e61847-08b7-4b3e-b342-8207d699fba2","Type":"ContainerDied","Data":"8f947eff379767c711c01f480c7d6541269a59fe350892893c7cb452bbae9380"}
Jul 14 01:44:58.152576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.152564    2578 scope.go:117] "RemoveContainer" containerID="40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76"
Jul 14 01:44:58.152874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.152854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"
Jul 14 01:44:58.163857 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.163800    2578 scope.go:117] "RemoveContainer" containerID="40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76"
Jul 14 01:44:58.164071 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:44:58.164053    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76\": container with ID starting with 40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76 not found: ID does not exist" containerID="40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76"
Jul 14 01:44:58.164120 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.164079    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76"} err="failed to get container status \"40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76\": rpc error: code = NotFound desc = could not find container \"40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76\": container with ID starting with 40cf4708a6bfb28f6580fe85fd9aa61ed703722ba81bce2351addc776ef8fd76 not found: ID does not exist"
Jul 14 01:44:58.174510 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.174486    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"]
Jul 14 01:44:58.179875 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.179855    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/sub-test-model-a21fee-kserve-67d794c95-dngtl"]
Jul 14 01:44:58.691698 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.691659    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e08baa-2150-46e1-9ffe-6d720d1ddc06" path="/var/lib/kubelet/pods/c2e08baa-2150-46e1-9ffe-6d720d1ddc06/volumes"
Jul 14 01:44:58.692088 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.692074    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e61847-08b7-4b3e-b342-8207d699fba2" path="/var/lib/kubelet/pods/c8e61847-08b7-4b3e-b342-8207d699fba2/volumes"
Jul 14 01:44:58.765149 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.765125    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:44:58.774207 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:44:58.774180    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:45:04.647877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:04.647838    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:45:04.647877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:04.647890    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:45:04.660016 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:04.659987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:45:05.197966 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:05.197933    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:45:08.435067 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:08.435032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:45:16.750759 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.750725    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"]
Jul 14 01:45:16.751249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.751159    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e61847-08b7-4b3e-b342-8207d699fba2" containerName="main"
Jul 14 01:45:16.751249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.751172    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e61847-08b7-4b3e-b342-8207d699fba2" containerName="main"
Jul 14 01:45:16.751249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.751184    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e08baa-2150-46e1-9ffe-6d720d1ddc06" containerName="main"
Jul 14 01:45:16.751249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.751189    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e08baa-2150-46e1-9ffe-6d720d1ddc06" containerName="main"
Jul 14 01:45:16.751249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.751249    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e61847-08b7-4b3e-b342-8207d699fba2" containerName="main"
Jul 14 01:45:16.751535 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.751259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2e08baa-2150-46e1-9ffe-6d720d1ddc06" containerName="main"
Jul 14 01:45:16.754464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.754443    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.757249 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.757218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"rate-test-model-a21fee-kserve-self-signed-certs\""
Jul 14 01:45:16.757865 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.757846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"kube-root-ca.crt\""
Jul 14 01:45:16.758005 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.757980    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"default-dockercfg-c69hq\""
Jul 14 01:45:16.758005 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.757997    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a21fee\"/\"openshift-service-ca.crt\""
Jul 14 01:45:16.776960 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.776931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"]
Jul 14 01:45:16.897161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.897129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f13955-be4d-4707-bc08-248eca46a801-tls-certs\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.897326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.897172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-dshm\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.897326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.897254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-tmp-dir\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.897326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.897286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-model-cache\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.897326 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.897319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-home\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.897526 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.897419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7k5\" (UniqueName: \"kubernetes.io/projected/c0f13955-be4d-4707-bc08-248eca46a801-kube-api-access-zk7k5\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998059 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7k5\" (UniqueName: \"kubernetes.io/projected/c0f13955-be4d-4707-bc08-248eca46a801-kube-api-access-zk7k5\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f13955-be4d-4707-bc08-248eca46a801-tls-certs\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-dshm\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-tmp-dir\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-model-cache\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-home\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998621 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-tmp-dir\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998749 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-home\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:16.998749 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:16.998678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-model-cache\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:17.000560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:17.000538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-dshm\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:17.000799 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:17.000757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f13955-be4d-4707-bc08-248eca46a801-tls-certs\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:17.005267 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:17.005245    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7k5\" (UniqueName: \"kubernetes.io/projected/c0f13955-be4d-4707-bc08-248eca46a801-kube-api-access-zk7k5\") pod \"rate-test-model-a21fee-kserve-5f68874667-lxg9p\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") " pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:17.063860 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:17.063834    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:17.197444 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:17.197413    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"]
Jul 14 01:45:17.200737 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:45:17.200710    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0f13955_be4d_4707_bc08_248eca46a801.slice/crio-76353803d4d7b96e0a939d0d063f6046822f8f09ad5c1d11cebab38ba27fd19d WatchSource:0}: Error finding container 76353803d4d7b96e0a939d0d063f6046822f8f09ad5c1d11cebab38ba27fd19d: Status 404 returned error can't find the container with id 76353803d4d7b96e0a939d0d063f6046822f8f09ad5c1d11cebab38ba27fd19d
Jul 14 01:45:17.248734 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:17.248665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p" event={"ID":"c0f13955-be4d-4707-bc08-248eca46a801","Type":"ContainerStarted","Data":"76353803d4d7b96e0a939d0d063f6046822f8f09ad5c1d11cebab38ba27fd19d"}
Jul 14 01:45:18.254917 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:18.254884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p" event={"ID":"c0f13955-be4d-4707-bc08-248eca46a801","Type":"ContainerStarted","Data":"aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f"}
Jul 14 01:45:18.271585 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:18.271535    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p" podStartSLOduration=2.27151907 podStartE2EDuration="2.27151907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:45:18.269756825 +0000 UTC m=+2720.157021455" watchObservedRunningTime="2026-07-14 01:45:18.27151907 +0000 UTC m=+2720.158783739"
Jul 14 01:45:24.139721 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:24.139668    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:45:27.064315 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:27.064282    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:27.064775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:27.064375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:27.076915 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:27.076893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:27.310230 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:27.310201    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:45:45.330245 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:45:45.330207    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:46:00.834124 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:00.834083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:46:06.840120 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:06.840088    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"]
Jul 14 01:46:07.615204 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:07.615157    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"]
Jul 14 01:46:12.169696 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.169651    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7366-140187612603696"]
Jul 14 01:46:12.173184 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.173169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:12.175279 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.175259    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-2vgml\""
Jul 14 01:46:12.178202 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.178177    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187612603696"]
Jul 14 01:46:12.285861 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.285836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6m8n\" (UniqueName: \"kubernetes.io/projected/6a529b28-cea5-4580-a5c6-f93256571c40-kube-api-access-x6m8n\") pod \"test-curl-7366-140187612603696\" (UID: \"6a529b28-cea5-4580-a5c6-f93256571c40\") " pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:12.386330 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.386304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6m8n\" (UniqueName: \"kubernetes.io/projected/6a529b28-cea5-4580-a5c6-f93256571c40-kube-api-access-x6m8n\") pod \"test-curl-7366-140187612603696\" (UID: \"6a529b28-cea5-4580-a5c6-f93256571c40\") " pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:12.393447 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.393423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6m8n\" (UniqueName: \"kubernetes.io/projected/6a529b28-cea5-4580-a5c6-f93256571c40-kube-api-access-x6m8n\") pod \"test-curl-7366-140187612603696\" (UID: \"6a529b28-cea5-4580-a5c6-f93256571c40\") " pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:12.482969 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.482904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:12.613406 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.613381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187612603696"]
Jul 14 01:46:12.615299 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:46:12.615272    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a529b28_cea5_4580_a5c6_f93256571c40.slice/crio-bdb20a73789214dd4c5463d70d493cc0ad25a950190ecf2e185d56753dfcf5f5 WatchSource:0}: Error finding container bdb20a73789214dd4c5463d70d493cc0ad25a950190ecf2e185d56753dfcf5f5: Status 404 returned error can't find the container with id bdb20a73789214dd4c5463d70d493cc0ad25a950190ecf2e185d56753dfcf5f5
Jul 14 01:46:12.616732 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:12.616712    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:46:13.533148 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:13.533110    2578 generic.go:356] "Generic (PLEG): container finished" podID="6a529b28-cea5-4580-a5c6-f93256571c40" containerID="9389a959a3756f0ae4704e7d68ec160164d8c748f88fc6748055e174a490a304" exitCode=0
Jul 14 01:46:13.533500 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:13.533184    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187612603696" event={"ID":"6a529b28-cea5-4580-a5c6-f93256571c40","Type":"ContainerDied","Data":"9389a959a3756f0ae4704e7d68ec160164d8c748f88fc6748055e174a490a304"}
Jul 14 01:46:13.533500 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:13.533212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187612603696" event={"ID":"6a529b28-cea5-4580-a5c6-f93256571c40","Type":"ContainerStarted","Data":"bdb20a73789214dd4c5463d70d493cc0ad25a950190ecf2e185d56753dfcf5f5"}
Jul 14 01:46:14.673165 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.673140    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:14.705106 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.705080    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6m8n\" (UniqueName: \"kubernetes.io/projected/6a529b28-cea5-4580-a5c6-f93256571c40-kube-api-access-x6m8n\") pod \"6a529b28-cea5-4580-a5c6-f93256571c40\" (UID: \"6a529b28-cea5-4580-a5c6-f93256571c40\") "
Jul 14 01:46:14.707332 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.707303    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a529b28-cea5-4580-a5c6-f93256571c40-kube-api-access-x6m8n" (OuterVolumeSpecName: "kube-api-access-x6m8n") pod "6a529b28-cea5-4580-a5c6-f93256571c40" (UID: "6a529b28-cea5-4580-a5c6-f93256571c40"). InnerVolumeSpecName "kube-api-access-x6m8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:14.708349 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.708328    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7366-140187612603696_6a529b28-cea5-4580-a5c6-f93256571c40/test-curl-7366-140187612603696/0.log"
Jul 14 01:46:14.730395 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.730374    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7366-140187612603696"]
Jul 14 01:46:14.735788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.735767    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7366-140187612603696"]
Jul 14 01:46:14.806602 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.806534    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6m8n\" (UniqueName: \"kubernetes.io/projected/6a529b28-cea5-4580-a5c6-f93256571c40-kube-api-access-x6m8n\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:14.899363 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.899333    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7366-140187616088112"]
Jul 14 01:46:14.899937 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.899910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a529b28-cea5-4580-a5c6-f93256571c40" containerName="test-curl-7366-140187612603696"
Jul 14 01:46:14.899937 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.899937    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a529b28-cea5-4580-a5c6-f93256571c40" containerName="test-curl-7366-140187612603696"
Jul 14 01:46:14.900094 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.900058    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a529b28-cea5-4580-a5c6-f93256571c40" containerName="test-curl-7366-140187612603696"
Jul 14 01:46:14.903281 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.903263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:14.907985 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.907959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrb4\" (UniqueName: \"kubernetes.io/projected/41ca4b0c-8bab-447b-a038-ea283578d68e-kube-api-access-qgrb4\") pod \"test-curl-7366-140187616088112\" (UID: \"41ca4b0c-8bab-447b-a038-ea283578d68e\") " pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:14.908098 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.908007    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187616088112"]
Jul 14 01:46:14.937739 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:14.937708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:46:15.008868 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.008835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrb4\" (UniqueName: \"kubernetes.io/projected/41ca4b0c-8bab-447b-a038-ea283578d68e-kube-api-access-qgrb4\") pod \"test-curl-7366-140187616088112\" (UID: \"41ca4b0c-8bab-447b-a038-ea283578d68e\") " pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:15.016295 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.016267    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrb4\" (UniqueName: \"kubernetes.io/projected/41ca4b0c-8bab-447b-a038-ea283578d68e-kube-api-access-qgrb4\") pod \"test-curl-7366-140187616088112\" (UID: \"41ca4b0c-8bab-447b-a038-ea283578d68e\") " pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:15.213806 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.213776    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:15.345113 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.345090    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187616088112"]
Jul 14 01:46:15.347164 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:46:15.347137    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41ca4b0c_8bab_447b_a038_ea283578d68e.slice/crio-fda0ef381e87d11fe53e075e5feb629708e5d19c7a9eddcf1baca8a0dc736ef2 WatchSource:0}: Error finding container fda0ef381e87d11fe53e075e5feb629708e5d19c7a9eddcf1baca8a0dc736ef2: Status 404 returned error can't find the container with id fda0ef381e87d11fe53e075e5feb629708e5d19c7a9eddcf1baca8a0dc736ef2
Jul 14 01:46:15.542813 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.542743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187616088112" event={"ID":"41ca4b0c-8bab-447b-a038-ea283578d68e","Type":"ContainerStarted","Data":"fda0ef381e87d11fe53e075e5feb629708e5d19c7a9eddcf1baca8a0dc736ef2"}
Jul 14 01:46:15.543877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.543860    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb20a73789214dd4c5463d70d493cc0ad25a950190ecf2e185d56753dfcf5f5"
Jul 14 01:46:15.543974 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:15.543892    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:15.553892 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:15.553870    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7366-140187612603696\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="6a529b28-cea5-4580-a5c6-f93256571c40" pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:16.550432 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:16.550398    2578 generic.go:356] "Generic (PLEG): container finished" podID="41ca4b0c-8bab-447b-a038-ea283578d68e" containerID="da58c6ad32735a5139af8296db908ecd5754498eda80beb425a7003224c14307" exitCode=0
Jul 14 01:46:16.550913 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:16.550484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187616088112" event={"ID":"41ca4b0c-8bab-447b-a038-ea283578d68e","Type":"ContainerDied","Data":"da58c6ad32735a5139af8296db908ecd5754498eda80beb425a7003224c14307"}
Jul 14 01:46:16.552576 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:16.552554    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7366-140187612603696\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="6a529b28-cea5-4580-a5c6-f93256571c40" pod="openshift-ingress/test-curl-7366-140187612603696"
Jul 14 01:46:16.691523 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:16.691492    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a529b28-cea5-4580-a5c6-f93256571c40" path="/var/lib/kubelet/pods/6a529b28-cea5-4580-a5c6-f93256571c40/volumes"
Jul 14 01:46:17.694900 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.694876    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:17.715504 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.715476    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7366-140187616088112_41ca4b0c-8bab-447b-a038-ea283578d68e/test-curl-7366-140187616088112/0.log"
Jul 14 01:46:17.732927 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.732897    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgrb4\" (UniqueName: \"kubernetes.io/projected/41ca4b0c-8bab-447b-a038-ea283578d68e-kube-api-access-qgrb4\") pod \"41ca4b0c-8bab-447b-a038-ea283578d68e\" (UID: \"41ca4b0c-8bab-447b-a038-ea283578d68e\") "
Jul 14 01:46:17.734978 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.734956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ca4b0c-8bab-447b-a038-ea283578d68e-kube-api-access-qgrb4" (OuterVolumeSpecName: "kube-api-access-qgrb4") pod "41ca4b0c-8bab-447b-a038-ea283578d68e" (UID: "41ca4b0c-8bab-447b-a038-ea283578d68e"). InnerVolumeSpecName "kube-api-access-qgrb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:17.737773 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.737752    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7366-140187616088112"]
Jul 14 01:46:17.742201 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.742183    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7366-140187616088112"]
Jul 14 01:46:17.833970 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:17.833908    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrb4\" (UniqueName: \"kubernetes.io/projected/41ca4b0c-8bab-447b-a038-ea283578d68e-kube-api-access-qgrb4\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:18.561702 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:18.561657    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fda0ef381e87d11fe53e075e5feb629708e5d19c7a9eddcf1baca8a0dc736ef2"
Jul 14 01:46:18.561702 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:18.561679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:18.570971 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:18.570943    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7366-140187616088112\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="41ca4b0c-8bab-447b-a038-ea283578d68e" pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:18.690243 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:18.690200    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7366-140187616088112\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="41ca4b0c-8bab-447b-a038-ea283578d68e" pod="openshift-ingress/test-curl-7366-140187616088112"
Jul 14 01:46:18.692255 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:18.692233    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ca4b0c-8bab-447b-a038-ea283578d68e" path="/var/lib/kubelet/pods/41ca4b0c-8bab-447b-a038-ea283578d68e/volumes"
Jul 14 01:46:20.200911 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.200877    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:20.201279 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.201253    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ca4b0c-8bab-447b-a038-ea283578d68e" containerName="test-curl-7366-140187616088112"
Jul 14 01:46:20.201279 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.201264    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ca4b0c-8bab-447b-a038-ea283578d68e" containerName="test-curl-7366-140187616088112"
Jul 14 01:46:20.201350 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.201331    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ca4b0c-8bab-447b-a038-ea283578d68e" containerName="test-curl-7366-140187616088112"
Jul 14 01:46:20.204625 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.204606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:20.206667 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.206647    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-2vgml\""
Jul 14 01:46:20.207992 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.207973    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:20.258122 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.258097    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf9vn\" (UniqueName: \"kubernetes.io/projected/9e51c456-cce4-4220-9b6e-9571f89a8565-kube-api-access-zf9vn\") pod \"test-curl-7366-140187616089120\" (UID: \"9e51c456-cce4-4220-9b6e-9571f89a8565\") " pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:20.358704 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.358663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf9vn\" (UniqueName: \"kubernetes.io/projected/9e51c456-cce4-4220-9b6e-9571f89a8565-kube-api-access-zf9vn\") pod \"test-curl-7366-140187616089120\" (UID: \"9e51c456-cce4-4220-9b6e-9571f89a8565\") " pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:20.366582 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.366549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf9vn\" (UniqueName: \"kubernetes.io/projected/9e51c456-cce4-4220-9b6e-9571f89a8565-kube-api-access-zf9vn\") pod \"test-curl-7366-140187616089120\" (UID: \"9e51c456-cce4-4220-9b6e-9571f89a8565\") " pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:20.515079 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.515018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:20.642858 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:20.642831    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:20.644582 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:46:20.644548    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e51c456_cce4_4220_9b6e_9571f89a8565.slice/crio-95cf73f4416e66f7e8d9ed6dfc0d1483aaf101c810a5250d0783423d67361601 WatchSource:0}: Error finding container 95cf73f4416e66f7e8d9ed6dfc0d1483aaf101c810a5250d0783423d67361601: Status 404 returned error can't find the container with id 95cf73f4416e66f7e8d9ed6dfc0d1483aaf101c810a5250d0783423d67361601
Jul 14 01:46:21.580780 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:21.580678    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e51c456-cce4-4220-9b6e-9571f89a8565" containerID="2f5a43b8bb3f5c1ff656b92fe0b3b4fb41057513ebb8df766d8edfbbd8ea462f" exitCode=0
Jul 14 01:46:21.581124 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:21.580778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187616089120" event={"ID":"9e51c456-cce4-4220-9b6e-9571f89a8565","Type":"ContainerDied","Data":"2f5a43b8bb3f5c1ff656b92fe0b3b4fb41057513ebb8df766d8edfbbd8ea462f"}
Jul 14 01:46:21.581124 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:21.580805    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187616089120" event={"ID":"9e51c456-cce4-4220-9b6e-9571f89a8565","Type":"ContainerStarted","Data":"95cf73f4416e66f7e8d9ed6dfc0d1483aaf101c810a5250d0783423d67361601"}
Jul 14 01:46:21.861805 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:21.861726    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm" podUID="7920cac1-cd14-4033-9320-05d208d6e470" containerName="main" containerID="cri-o://985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d" gracePeriod=45
Jul 14 01:46:22.113415 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.113355    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:46:22.174062 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7920cac1-cd14-4033-9320-05d208d6e470-tls-certs\") pod \"7920cac1-cd14-4033-9320-05d208d6e470\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") "
Jul 14 01:46:22.174206 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174070    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-model-cache\") pod \"7920cac1-cd14-4033-9320-05d208d6e470\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") "
Jul 14 01:46:22.174206 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174099    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-dshm\") pod \"7920cac1-cd14-4033-9320-05d208d6e470\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") "
Jul 14 01:46:22.174206 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-home\") pod \"7920cac1-cd14-4033-9320-05d208d6e470\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") "
Jul 14 01:46:22.174206 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174176    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-tmp-dir\") pod \"7920cac1-cd14-4033-9320-05d208d6e470\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") "
Jul 14 01:46:22.174375 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp5qp\" (UniqueName: \"kubernetes.io/projected/7920cac1-cd14-4033-9320-05d208d6e470-kube-api-access-vp5qp\") pod \"7920cac1-cd14-4033-9320-05d208d6e470\" (UID: \"7920cac1-cd14-4033-9320-05d208d6e470\") "
Jul 14 01:46:22.174375 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174320    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-model-cache" (OuterVolumeSpecName: "model-cache") pod "7920cac1-cd14-4033-9320-05d208d6e470" (UID: "7920cac1-cd14-4033-9320-05d208d6e470"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.174466 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174386    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-home" (OuterVolumeSpecName: "home") pod "7920cac1-cd14-4033-9320-05d208d6e470" (UID: "7920cac1-cd14-4033-9320-05d208d6e470"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.174527 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174487    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7920cac1-cd14-4033-9320-05d208d6e470" (UID: "7920cac1-cd14-4033-9320-05d208d6e470"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.174576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174561    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.174627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174575    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.174627 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.174583    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.176354 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.176328    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-dshm" (OuterVolumeSpecName: "dshm") pod "7920cac1-cd14-4033-9320-05d208d6e470" (UID: "7920cac1-cd14-4033-9320-05d208d6e470"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.176464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.176367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7920cac1-cd14-4033-9320-05d208d6e470-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7920cac1-cd14-4033-9320-05d208d6e470" (UID: "7920cac1-cd14-4033-9320-05d208d6e470"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:46:22.176464 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.176419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7920cac1-cd14-4033-9320-05d208d6e470-kube-api-access-vp5qp" (OuterVolumeSpecName: "kube-api-access-vp5qp") pod "7920cac1-cd14-4033-9320-05d208d6e470" (UID: "7920cac1-cd14-4033-9320-05d208d6e470"). InnerVolumeSpecName "kube-api-access-vp5qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:22.275336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.275305    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5qp\" (UniqueName: \"kubernetes.io/projected/7920cac1-cd14-4033-9320-05d208d6e470-kube-api-access-vp5qp\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.275336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.275331    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7920cac1-cd14-4033-9320-05d208d6e470-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.275336 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.275343    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7920cac1-cd14-4033-9320-05d208d6e470-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.587139 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.587104    2578 generic.go:356] "Generic (PLEG): container finished" podID="7920cac1-cd14-4033-9320-05d208d6e470" containerID="985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d" exitCode=0
Jul 14 01:46:22.587560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.587148    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"
Jul 14 01:46:22.587560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.587193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm" event={"ID":"7920cac1-cd14-4033-9320-05d208d6e470","Type":"ContainerDied","Data":"985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d"}
Jul 14 01:46:22.587560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.587233    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm" event={"ID":"7920cac1-cd14-4033-9320-05d208d6e470","Type":"ContainerDied","Data":"7744575d72b4a2bd5af6e3cd2fa23d034b787d9ff917337432d003152ebf58f5"}
Jul 14 01:46:22.587560 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.587251    2578 scope.go:117] "RemoveContainer" containerID="985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d"
Jul 14 01:46:22.596921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.596904    2578 scope.go:117] "RemoveContainer" containerID="985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d"
Jul 14 01:46:22.597191 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:22.597169    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d\": container with ID starting with 985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d not found: ID does not exist" containerID="985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d"
Jul 14 01:46:22.597239 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.597203    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d"} err="failed to get container status \"985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d\": rpc error: code = NotFound desc = could not find container \"985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d\": container with ID starting with 985778eac2865fcb9660697d5b9141d37f44002db863a5238de104ed17438e4d not found: ID does not exist"
Jul 14 01:46:22.609089 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.609066    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"]
Jul 14 01:46:22.614493 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.614468    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c6543c/rate-test-model-c6543c-kserve-5bb7c6955-b8vcm"]
Jul 14 01:46:22.635944 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.635874    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p" podUID="c0f13955-be4d-4707-bc08-248eca46a801" containerName="main" containerID="cri-o://aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f" gracePeriod=45
Jul 14 01:46:22.692105 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.692075    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7920cac1-cd14-4033-9320-05d208d6e470" path="/var/lib/kubelet/pods/7920cac1-cd14-4033-9320-05d208d6e470/volumes"
Jul 14 01:46:22.758874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.758849    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:22.778559 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.778537    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7366-140187616089120_9e51c456-cce4-4220-9b6e-9571f89a8565/test-curl-7366-140187616089120/0.log"
Jul 14 01:46:22.779727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.779710    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf9vn\" (UniqueName: \"kubernetes.io/projected/9e51c456-cce4-4220-9b6e-9571f89a8565-kube-api-access-zf9vn\") pod \"9e51c456-cce4-4220-9b6e-9571f89a8565\" (UID: \"9e51c456-cce4-4220-9b6e-9571f89a8565\") "
Jul 14 01:46:22.782904 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.782874    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e51c456-cce4-4220-9b6e-9571f89a8565-kube-api-access-zf9vn" (OuterVolumeSpecName: "kube-api-access-zf9vn") pod "9e51c456-cce4-4220-9b6e-9571f89a8565" (UID: "9e51c456-cce4-4220-9b6e-9571f89a8565"). InnerVolumeSpecName "kube-api-access-zf9vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:22.802417 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.802387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:22.806996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.806974    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:22.879994 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.879974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:46:22.880452 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.880433    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf9vn\" (UniqueName: \"kubernetes.io/projected/9e51c456-cce4-4220-9b6e-9571f89a8565-kube-api-access-zf9vn\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.926457 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926434    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:22.926825 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926812    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0f13955-be4d-4707-bc08-248eca46a801" containerName="main"
Jul 14 01:46:22.926886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926827    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0f13955-be4d-4707-bc08-248eca46a801" containerName="main"
Jul 14 01:46:22.926886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926843    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7920cac1-cd14-4033-9320-05d208d6e470" containerName="main"
Jul 14 01:46:22.926886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926848    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7920cac1-cd14-4033-9320-05d208d6e470" containerName="main"
Jul 14 01:46:22.926886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e51c456-cce4-4220-9b6e-9571f89a8565" containerName="test-curl-7366-140187616089120"
Jul 14 01:46:22.926886 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926863    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e51c456-cce4-4220-9b6e-9571f89a8565" containerName="test-curl-7366-140187616089120"
Jul 14 01:46:22.927055 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926923    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7920cac1-cd14-4033-9320-05d208d6e470" containerName="main"
Jul 14 01:46:22.927055 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926931    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e51c456-cce4-4220-9b6e-9571f89a8565" containerName="test-curl-7366-140187616089120"
Jul 14 01:46:22.927055 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.926938    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0f13955-be4d-4707-bc08-248eca46a801" containerName="main"
Jul 14 01:46:22.929798 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.929785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:22.932873 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.932853    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:22.981396 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f13955-be4d-4707-bc08-248eca46a801-tls-certs\") pod \"c0f13955-be4d-4707-bc08-248eca46a801\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") "
Jul 14 01:46:22.981513 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981428    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-model-cache\") pod \"c0f13955-be4d-4707-bc08-248eca46a801\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") "
Jul 14 01:46:22.981513 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981454    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk7k5\" (UniqueName: \"kubernetes.io/projected/c0f13955-be4d-4707-bc08-248eca46a801-kube-api-access-zk7k5\") pod \"c0f13955-be4d-4707-bc08-248eca46a801\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") "
Jul 14 01:46:22.981513 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981507    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-home\") pod \"c0f13955-be4d-4707-bc08-248eca46a801\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") "
Jul 14 01:46:22.981665 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981534    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-tmp-dir\") pod \"c0f13955-be4d-4707-bc08-248eca46a801\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") "
Jul 14 01:46:22.981665 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981617    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-dshm\") pod \"c0f13955-be4d-4707-bc08-248eca46a801\" (UID: \"c0f13955-be4d-4707-bc08-248eca46a801\") "
Jul 14 01:46:22.981791 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0f13955-be4d-4707-bc08-248eca46a801" (UID: "c0f13955-be4d-4707-bc08-248eca46a801"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.981791 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-home" (OuterVolumeSpecName: "home") pod "c0f13955-be4d-4707-bc08-248eca46a801" (UID: "c0f13955-be4d-4707-bc08-248eca46a801"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.981917 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0f13955-be4d-4707-bc08-248eca46a801" (UID: "c0f13955-be4d-4707-bc08-248eca46a801"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:22.981968 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwxg\" (UniqueName: \"kubernetes.io/projected/aa796ca6-9b04-4e66-b68a-9a84444f788e-kube-api-access-4dwxg\") pod \"test-curl-7366-140187616089120\" (UID: \"aa796ca6-9b04-4e66-b68a-9a84444f788e\") " pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:22.982026 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.981989    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-model-cache\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.982026 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.982008    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-home\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.982026 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.982020    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-tmp-dir\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:22.983609 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.983590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0f13955-be4d-4707-bc08-248eca46a801-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0f13955-be4d-4707-bc08-248eca46a801" (UID: "c0f13955-be4d-4707-bc08-248eca46a801"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:46:22.983713 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.983633    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0f13955-be4d-4707-bc08-248eca46a801-kube-api-access-zk7k5" (OuterVolumeSpecName: "kube-api-access-zk7k5") pod "c0f13955-be4d-4707-bc08-248eca46a801" (UID: "c0f13955-be4d-4707-bc08-248eca46a801"). InnerVolumeSpecName "kube-api-access-zk7k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:22.983713 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:22.983641    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-dshm" (OuterVolumeSpecName: "dshm") pod "c0f13955-be4d-4707-bc08-248eca46a801" (UID: "c0f13955-be4d-4707-bc08-248eca46a801"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:46:23.082775 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.082750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwxg\" (UniqueName: \"kubernetes.io/projected/aa796ca6-9b04-4e66-b68a-9a84444f788e-kube-api-access-4dwxg\") pod \"test-curl-7366-140187616089120\" (UID: \"aa796ca6-9b04-4e66-b68a-9a84444f788e\") " pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:23.082874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.082817    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0f13955-be4d-4707-bc08-248eca46a801-tls-certs\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:23.082874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.082827    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7k5\" (UniqueName: \"kubernetes.io/projected/c0f13955-be4d-4707-bc08-248eca46a801-kube-api-access-zk7k5\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:23.082874 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.082837    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0f13955-be4d-4707-bc08-248eca46a801-dshm\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:23.090101 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.090075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwxg\" (UniqueName: \"kubernetes.io/projected/aa796ca6-9b04-4e66-b68a-9a84444f788e-kube-api-access-4dwxg\") pod \"test-curl-7366-140187616089120\" (UID: \"aa796ca6-9b04-4e66-b68a-9a84444f788e\") " pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:23.240755 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.240729    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:23.368845 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.368816    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:23.371842 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:46:23.371814    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa796ca6_9b04_4e66_b68a_9a84444f788e.slice/crio-1762280aed3b9966f29bed6ed7aebebb32d31074f996a66ae2fee6004786730d WatchSource:0}: Error finding container 1762280aed3b9966f29bed6ed7aebebb32d31074f996a66ae2fee6004786730d: Status 404 returned error can't find the container with id 1762280aed3b9966f29bed6ed7aebebb32d31074f996a66ae2fee6004786730d
Jul 14 01:46:23.592859 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.592759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187616089120" event={"ID":"aa796ca6-9b04-4e66-b68a-9a84444f788e","Type":"ContainerStarted","Data":"1762280aed3b9966f29bed6ed7aebebb32d31074f996a66ae2fee6004786730d"}
Jul 14 01:46:23.594107 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.594082    2578 generic.go:356] "Generic (PLEG): container finished" podID="c0f13955-be4d-4707-bc08-248eca46a801" containerID="aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f" exitCode=0
Jul 14 01:46:23.594231 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.594131    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"
Jul 14 01:46:23.594231 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.594168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p" event={"ID":"c0f13955-be4d-4707-bc08-248eca46a801","Type":"ContainerDied","Data":"aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f"}
Jul 14 01:46:23.594231 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.594208    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p" event={"ID":"c0f13955-be4d-4707-bc08-248eca46a801","Type":"ContainerDied","Data":"76353803d4d7b96e0a939d0d063f6046822f8f09ad5c1d11cebab38ba27fd19d"}
Jul 14 01:46:23.594231 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.594226    2578 scope.go:117] "RemoveContainer" containerID="aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f"
Jul 14 01:46:23.596539 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.596522    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95cf73f4416e66f7e8d9ed6dfc0d1483aaf101c810a5250d0783423d67361601"
Jul 14 01:46:23.596663 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.596560    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:23.604516 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.604496    2578 scope.go:117] "RemoveContainer" containerID="aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f"
Jul 14 01:46:23.604834 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:23.604808    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f\": container with ID starting with aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f not found: ID does not exist" containerID="aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f"
Jul 14 01:46:23.604945 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.604844    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f"} err="failed to get container status \"aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f\": rpc error: code = NotFound desc = could not find container \"aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f\": container with ID starting with aa87d495c6cdcc98bbc7439a21d890d16971ac886c2f30e5836fb0de9c6f610f not found: ID does not exist"
Jul 14 01:46:23.616136 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.616112    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"]
Jul 14 01:46:23.619840 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.619819    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a21fee/rate-test-model-a21fee-kserve-5f68874667-lxg9p"]
Jul 14 01:46:23.620204 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:23.620182    2578 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7366-140187616089120" oldPodUID="9e51c456-cce4-4220-9b6e-9571f89a8565" podUID="aa796ca6-9b04-4e66-b68a-9a84444f788e"
Jul 14 01:46:24.604214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:24.604185    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa796ca6-9b04-4e66-b68a-9a84444f788e" containerID="6b71399459bf6ba92697c48211ad53f8034b5b5b381602453801c316bb1e3545" exitCode=0
Jul 14 01:46:24.604214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:24.604218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7366-140187616089120" event={"ID":"aa796ca6-9b04-4e66-b68a-9a84444f788e","Type":"ContainerDied","Data":"6b71399459bf6ba92697c48211ad53f8034b5b5b381602453801c316bb1e3545"}
Jul 14 01:46:24.690422 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:24.690392    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e51c456-cce4-4220-9b6e-9571f89a8565" path="/var/lib/kubelet/pods/9e51c456-cce4-4220-9b6e-9571f89a8565/volumes"
Jul 14 01:46:24.690799 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:24.690786    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0f13955-be4d-4707-bc08-248eca46a801" path="/var/lib/kubelet/pods/c0f13955-be4d-4707-bc08-248eca46a801/volumes"
Jul 14 01:46:25.758222 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.758192    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:25.776518 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.776492    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7366-140187616089120_aa796ca6-9b04-4e66-b68a-9a84444f788e/test-curl-7366-140187616089120/0.log"
Jul 14 01:46:25.801813 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.801761    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:25.803572 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.803544    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7366-140187616089120"]
Jul 14 01:46:25.806803 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.806773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dwxg\" (UniqueName: \"kubernetes.io/projected/aa796ca6-9b04-4e66-b68a-9a84444f788e-kube-api-access-4dwxg\") pod \"aa796ca6-9b04-4e66-b68a-9a84444f788e\" (UID: \"aa796ca6-9b04-4e66-b68a-9a84444f788e\") "
Jul 14 01:46:25.808921 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.808894    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa796ca6-9b04-4e66-b68a-9a84444f788e-kube-api-access-4dwxg" (OuterVolumeSpecName: "kube-api-access-4dwxg") pod "aa796ca6-9b04-4e66-b68a-9a84444f788e" (UID: "aa796ca6-9b04-4e66-b68a-9a84444f788e"). InnerVolumeSpecName "kube-api-access-4dwxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:25.908566 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:25.908490    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwxg\" (UniqueName: \"kubernetes.io/projected/aa796ca6-9b04-4e66-b68a-9a84444f788e-kube-api-access-4dwxg\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:26.618224 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:26.618197    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1762280aed3b9966f29bed6ed7aebebb32d31074f996a66ae2fee6004786730d"
Jul 14 01:46:26.618224 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:26.618223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:26.627593 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:26.627568    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7366-140187616089120\" is forbidden: User \"system:node:ip-10-0-138-193.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-193.ec2.internal' and this object" podUID="aa796ca6-9b04-4e66-b68a-9a84444f788e" pod="openshift-ingress/test-curl-7366-140187616089120"
Jul 14 01:46:26.690834 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:26.690808    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa796ca6-9b04-4e66-b68a-9a84444f788e" path="/var/lib/kubelet/pods/aa796ca6-9b04-4e66-b68a-9a84444f788e/volumes"
Jul 14 01:46:36.369817 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.369781    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"]
Jul 14 01:46:36.370255 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.370066    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" podUID="569740af-a93b-4f26-b3b2-7944a77b9af7" containerName="maas-api" containerID="cri-o://44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498" gracePeriod=30
Jul 14 01:46:36.613561 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.613539    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:46:36.665899 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.665869    2578 generic.go:356] "Generic (PLEG): container finished" podID="569740af-a93b-4f26-b3b2-7944a77b9af7" containerID="44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498" exitCode=0
Jul 14 01:46:36.666035 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.665915    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"
Jul 14 01:46:36.666035 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.665952    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" event={"ID":"569740af-a93b-4f26-b3b2-7944a77b9af7","Type":"ContainerDied","Data":"44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498"}
Jul 14 01:46:36.666035 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.666001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz" event={"ID":"569740af-a93b-4f26-b3b2-7944a77b9af7","Type":"ContainerDied","Data":"0b307e9ecd34af31123e32cf7bbab54622bb3477342c953b5b3c05f28f3f2846"}
Jul 14 01:46:36.666035 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.666023    2578 scope.go:117] "RemoveContainer" containerID="44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498"
Jul 14 01:46:36.674404 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.674390    2578 scope.go:117] "RemoveContainer" containerID="44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498"
Jul 14 01:46:36.674656 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:46:36.674638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498\": container with ID starting with 44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498 not found: ID does not exist" containerID="44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498"
Jul 14 01:46:36.674743 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.674664    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498"} err="failed to get container status \"44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498\": rpc error: code = NotFound desc = could not find container \"44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498\": container with ID starting with 44d869e6fce160924e5dc62c7feb03fb5d40ac5d3d7f1255f37871631c8fd498 not found: ID does not exist"
Jul 14 01:46:36.702739 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.702721    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls\") pod \"569740af-a93b-4f26-b3b2-7944a77b9af7\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") "
Jul 14 01:46:36.702852 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.702780    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569740af-a93b-4f26-b3b2-7944a77b9af7-openshift-service-ca\") pod \"569740af-a93b-4f26-b3b2-7944a77b9af7\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") "
Jul 14 01:46:36.702852 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.702807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zk82\" (UniqueName: \"kubernetes.io/projected/569740af-a93b-4f26-b3b2-7944a77b9af7-kube-api-access-2zk82\") pod \"569740af-a93b-4f26-b3b2-7944a77b9af7\" (UID: \"569740af-a93b-4f26-b3b2-7944a77b9af7\") "
Jul 14 01:46:36.703140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.703117    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569740af-a93b-4f26-b3b2-7944a77b9af7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "569740af-a93b-4f26-b3b2-7944a77b9af7" (UID: "569740af-a93b-4f26-b3b2-7944a77b9af7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:46:36.704932 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.704911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569740af-a93b-4f26-b3b2-7944a77b9af7-kube-api-access-2zk82" (OuterVolumeSpecName: "kube-api-access-2zk82") pod "569740af-a93b-4f26-b3b2-7944a77b9af7" (UID: "569740af-a93b-4f26-b3b2-7944a77b9af7"). InnerVolumeSpecName "kube-api-access-2zk82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:46:36.705019 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.704962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "569740af-a93b-4f26-b3b2-7944a77b9af7" (UID: "569740af-a93b-4f26-b3b2-7944a77b9af7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:46:36.803878 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.803665    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569740af-a93b-4f26-b3b2-7944a77b9af7-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:36.803878 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.803724    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569740af-a93b-4f26-b3b2-7944a77b9af7-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:36.803878 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.803739    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zk82\" (UniqueName: \"kubernetes.io/projected/569740af-a93b-4f26-b3b2-7944a77b9af7-kube-api-access-2zk82\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:46:36.986505 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.986469    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"]
Jul 14 01:46:36.989802 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:36.989774    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c6543c-564cbb88c8-fpvkz"]
Jul 14 01:46:38.697020 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:38.696976    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569740af-a93b-4f26-b3b2-7944a77b9af7" path="/var/lib/kubelet/pods/569740af-a93b-4f26-b3b2-7944a77b9af7/volumes"
Jul 14 01:46:39.830214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:46:39.830179    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:47:02.701107 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:02.701074    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"]
Jul 14 01:47:02.701600 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:02.701293    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx" podUID="5a31e0ee-e8e9-408f-b55d-50160fe8a436" containerName="maas-api" containerID="cri-o://fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255" gracePeriod=30
Jul 14 01:47:03.139877 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.139857    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:47:03.238201 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.238117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31e0ee-e8e9-408f-b55d-50160fe8a436-maas-api-tls\") pod \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") "
Jul 14 01:47:03.238363 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.238246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc4wx\" (UniqueName: \"kubernetes.io/projected/5a31e0ee-e8e9-408f-b55d-50160fe8a436-kube-api-access-pc4wx\") pod \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") "
Jul 14 01:47:03.238363 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.238283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31e0ee-e8e9-408f-b55d-50160fe8a436-openshift-service-ca\") pod \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\" (UID: \"5a31e0ee-e8e9-408f-b55d-50160fe8a436\") "
Jul 14 01:47:03.238658 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.238622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a31e0ee-e8e9-408f-b55d-50160fe8a436-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a31e0ee-e8e9-408f-b55d-50160fe8a436" (UID: "5a31e0ee-e8e9-408f-b55d-50160fe8a436"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:47:03.240229 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.240203    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a31e0ee-e8e9-408f-b55d-50160fe8a436-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a31e0ee-e8e9-408f-b55d-50160fe8a436" (UID: "5a31e0ee-e8e9-408f-b55d-50160fe8a436"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:47:03.240410 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.240388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a31e0ee-e8e9-408f-b55d-50160fe8a436-kube-api-access-pc4wx" (OuterVolumeSpecName: "kube-api-access-pc4wx") pod "5a31e0ee-e8e9-408f-b55d-50160fe8a436" (UID: "5a31e0ee-e8e9-408f-b55d-50160fe8a436"). InnerVolumeSpecName "kube-api-access-pc4wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:47:03.339074 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.339035    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4wx\" (UniqueName: \"kubernetes.io/projected/5a31e0ee-e8e9-408f-b55d-50160fe8a436-kube-api-access-pc4wx\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:47:03.339074 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.339067    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a31e0ee-e8e9-408f-b55d-50160fe8a436-openshift-service-ca\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:47:03.339074 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.339076    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a31e0ee-e8e9-408f-b55d-50160fe8a436-maas-api-tls\") on node \"ip-10-0-138-193.ec2.internal\" DevicePath \"\""
Jul 14 01:47:03.796105 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.796068    2578 generic.go:356] "Generic (PLEG): container finished" podID="5a31e0ee-e8e9-408f-b55d-50160fe8a436" containerID="fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255" exitCode=0
Jul 14 01:47:03.796576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.796124    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"
Jul 14 01:47:03.796576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.796130    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx" event={"ID":"5a31e0ee-e8e9-408f-b55d-50160fe8a436","Type":"ContainerDied","Data":"fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255"}
Jul 14 01:47:03.796576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.796240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx" event={"ID":"5a31e0ee-e8e9-408f-b55d-50160fe8a436","Type":"ContainerDied","Data":"9091315113ca2f0faab6b8720d5c7e6dd712e36d6d54f6ec5f548e420cde2399"}
Jul 14 01:47:03.796576 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.796264    2578 scope.go:117] "RemoveContainer" containerID="fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255"
Jul 14 01:47:03.805049 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.805029    2578 scope.go:117] "RemoveContainer" containerID="fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255"
Jul 14 01:47:03.805296 ip-10-0-138-193 kubenswrapper[2578]: E0714 01:47:03.805280    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255\": container with ID starting with fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255 not found: ID does not exist" containerID="fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255"
Jul 14 01:47:03.805357 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.805301    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255"} err="failed to get container status \"fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255\": rpc error: code = NotFound desc = could not find container \"fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255\": container with ID starting with fe0f9b2a1b8eaedf54ce5d6dabdea8f818bb2edd8869848cfe3586f7d1d9d255 not found: ID does not exist"
Jul 14 01:47:03.816960 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.816940    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"]
Jul 14 01:47:03.820492 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:03.820472    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a21fee-75bf5cdb85-rxzxx"]
Jul 14 01:47:04.692443 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:04.692407    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a31e0ee-e8e9-408f-b55d-50160fe8a436" path="/var/lib/kubelet/pods/5a31e0ee-e8e9-408f-b55d-50160fe8a436/volumes"
Jul 14 01:47:05.831238 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:05.831205    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xpdt7"]
Jul 14 01:47:24.790531 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:24.790501    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554b6949c8-6dqbl_721178c4-28cb-40a6-b4dc-45407dafbd9b/copy-manifests/0.log"
Jul 14 01:47:24.803387 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:24.803365    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554b6949c8-6dqbl_721178c4-28cb-40a6-b4dc-45407dafbd9b/manager/0.log"
Jul 14 01:47:26.236485 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:26.236458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-ef50e2-7xpsd_29f45d67-6a19-40a0-9ed2-533187f733cc/revoke-keys/0.log"
Jul 14 01:47:26.367214 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:26.367184    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-99bb7c-b8dl5_978dde70-998d-477e-a63e-c72cf2f47f94/revoke-keys/0.log"
Jul 14 01:47:26.500530 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:26.500466    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-d3208c92-dg49w_eb734f39-eb62-48d9-99e1-4df4ff524181/revoke-keys/0.log"
Jul 14 01:47:29.734791 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:29.734759    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xpdt7_080da3cc-39fd-4eed-80e4-7c37429ccbe2/limitador/0.log"
Jul 14 01:47:29.854996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:29.854967    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nbhbd_abfa3bfb-585d-4c34-aae2-fe93c372cb69/manager/0.log"
Jul 14 01:47:30.469939 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:30.469908    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d956c5b-pdkbg_fb370852-4001-4d24-aebe-063138ee8c2c/kube-auth-proxy/0.log"
Jul 14 01:47:32.148607 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:32.148579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-49vvj_8d375ca4-2afa-4099-9458-fc4834f38a39/main/0.log"
Jul 14 01:47:36.316626 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.316586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrhh6/must-gather-xs7w9"]
Jul 14 01:47:36.317187 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317166    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569740af-a93b-4f26-b3b2-7944a77b9af7" containerName="maas-api"
Jul 14 01:47:36.317272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="569740af-a93b-4f26-b3b2-7944a77b9af7" containerName="maas-api"
Jul 14 01:47:36.317272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317209    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a31e0ee-e8e9-408f-b55d-50160fe8a436" containerName="maas-api"
Jul 14 01:47:36.317272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317217    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a31e0ee-e8e9-408f-b55d-50160fe8a436" containerName="maas-api"
Jul 14 01:47:36.317272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317248    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa796ca6-9b04-4e66-b68a-9a84444f788e" containerName="test-curl-7366-140187616089120"
Jul 14 01:47:36.317272 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317259    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa796ca6-9b04-4e66-b68a-9a84444f788e" containerName="test-curl-7366-140187616089120"
Jul 14 01:47:36.317519 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317343    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a31e0ee-e8e9-408f-b55d-50160fe8a436" containerName="maas-api"
Jul 14 01:47:36.317519 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317357    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="569740af-a93b-4f26-b3b2-7944a77b9af7" containerName="maas-api"
Jul 14 01:47:36.317519 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.317367    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa796ca6-9b04-4e66-b68a-9a84444f788e" containerName="test-curl-7366-140187616089120"
Jul 14 01:47:36.320334 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.320314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.322739 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.322718    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrhh6\"/\"openshift-service-ca.crt\""
Jul 14 01:47:36.323471 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.323456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lrhh6\"/\"default-dockercfg-cbk22\""
Jul 14 01:47:36.323528 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.323503    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lrhh6\"/\"kube-root-ca.crt\""
Jul 14 01:47:36.327965 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.327944    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrhh6/must-gather-xs7w9"]
Jul 14 01:47:36.432788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.432761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e0ceb0e-965a-4f5c-8d8f-08726748761d-must-gather-output\") pod \"must-gather-xs7w9\" (UID: \"1e0ceb0e-965a-4f5c-8d8f-08726748761d\") " pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.432788 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.432789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9l7\" (UniqueName: \"kubernetes.io/projected/1e0ceb0e-965a-4f5c-8d8f-08726748761d-kube-api-access-sf9l7\") pod \"must-gather-xs7w9\" (UID: \"1e0ceb0e-965a-4f5c-8d8f-08726748761d\") " pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.533591 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.533566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e0ceb0e-965a-4f5c-8d8f-08726748761d-must-gather-output\") pod \"must-gather-xs7w9\" (UID: \"1e0ceb0e-965a-4f5c-8d8f-08726748761d\") " pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.533591 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.533595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9l7\" (UniqueName: \"kubernetes.io/projected/1e0ceb0e-965a-4f5c-8d8f-08726748761d-kube-api-access-sf9l7\") pod \"must-gather-xs7w9\" (UID: \"1e0ceb0e-965a-4f5c-8d8f-08726748761d\") " pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.533914 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.533893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1e0ceb0e-965a-4f5c-8d8f-08726748761d-must-gather-output\") pod \"must-gather-xs7w9\" (UID: \"1e0ceb0e-965a-4f5c-8d8f-08726748761d\") " pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.541530 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.541506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9l7\" (UniqueName: \"kubernetes.io/projected/1e0ceb0e-965a-4f5c-8d8f-08726748761d-kube-api-access-sf9l7\") pod \"must-gather-xs7w9\" (UID: \"1e0ceb0e-965a-4f5c-8d8f-08726748761d\") " pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.630563 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.630499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrhh6/must-gather-xs7w9"
Jul 14 01:47:36.758599 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.758571    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrhh6/must-gather-xs7w9"]
Jul 14 01:47:36.760677 ip-10-0-138-193 kubenswrapper[2578]: W0714 01:47:36.760650    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e0ceb0e_965a_4f5c_8d8f_08726748761d.slice/crio-d736552ef836ef42fedf7b2ebb7f27770da4fe0d861ce081b5067decd0279895 WatchSource:0}: Error finding container d736552ef836ef42fedf7b2ebb7f27770da4fe0d861ce081b5067decd0279895: Status 404 returned error can't find the container with id d736552ef836ef42fedf7b2ebb7f27770da4fe0d861ce081b5067decd0279895
Jul 14 01:47:36.957182 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:36.957152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrhh6/must-gather-xs7w9" event={"ID":"1e0ceb0e-965a-4f5c-8d8f-08726748761d","Type":"ContainerStarted","Data":"d736552ef836ef42fedf7b2ebb7f27770da4fe0d861ce081b5067decd0279895"}
Jul 14 01:47:37.966700 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:37.966649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrhh6/must-gather-xs7w9" event={"ID":"1e0ceb0e-965a-4f5c-8d8f-08726748761d","Type":"ContainerStarted","Data":"7e2fdf408c4e3cafccabd3558926423cd8f2f9b9d113194e59035343aec09431"}
Jul 14 01:47:37.967079 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:37.966728    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrhh6/must-gather-xs7w9" event={"ID":"1e0ceb0e-965a-4f5c-8d8f-08726748761d","Type":"ContainerStarted","Data":"7dc2256ea22c9d8824db8a4fac163bab79f51d271a20c855d08302c04f239950"}
Jul 14 01:47:37.982044 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:37.981983    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrhh6/must-gather-xs7w9" podStartSLOduration=1.22210441 podStartE2EDuration="1.981962465s" totalImagesPullingTime="759.858055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:47:37.981583077 +0000 UTC m=+2859.868847706" watchObservedRunningTime="2026-07-14 01:47:37.981962465 +0000 UTC m=+2859.869227096"
Jul 14 01:47:39.044383 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:39.044358    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kt79b_447074a6-a97f-4e95-a5d4-3768f6929ba0/global-pull-secret-syncer/0.log"
Jul 14 01:47:39.150642 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:39.150616    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-djfrp_1e29c106-2406-4b8b-8a35-a50c782e0cb2/konnectivity-agent/0.log"
Jul 14 01:47:39.218496 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:39.218451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-193.ec2.internal_9040b7d4a25b23ceee331d8a28092737/haproxy/0.log"
Jul 14 01:47:44.091727 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:44.091632    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xpdt7_080da3cc-39fd-4eed-80e4-7c37429ccbe2/limitador/0.log"
Jul 14 01:47:44.176883 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:44.176812    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nbhbd_abfa3bfb-585d-4c34-aae2-fe93c372cb69/manager/0.log"
Jul 14 01:47:45.889140 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:45.889102    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/alertmanager/0.log"
Jul 14 01:47:45.911900 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:45.911876    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/config-reloader/0.log"
Jul 14 01:47:45.932906 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:45.932878    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/kube-rbac-proxy-web/0.log"
Jul 14 01:47:45.954217 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:45.954187    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/kube-rbac-proxy/0.log"
Jul 14 01:47:45.977822 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:45.977790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/kube-rbac-proxy-metric/0.log"
Jul 14 01:47:46.007052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.007022    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/prom-label-proxy/0.log"
Jul 14 01:47:46.034802 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.034774    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a5909f41-52a5-4a5b-8220-f00bd9c71fe5/init-config-reloader/0.log"
Jul 14 01:47:46.095269 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.095238    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4kjk2_c741dbc7-4e9c-4677-8f5c-ba80b9804ed7/kube-state-metrics/0.log"
Jul 14 01:47:46.110737 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.110712    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4kjk2_c741dbc7-4e9c-4677-8f5c-ba80b9804ed7/kube-rbac-proxy-main/0.log"
Jul 14 01:47:46.130885 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.130856    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-4kjk2_c741dbc7-4e9c-4677-8f5c-ba80b9804ed7/kube-rbac-proxy-self/0.log"
Jul 14 01:47:46.178881 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.178853    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-645c9f5b98-p4wr9_b4679efc-70e8-4162-a5b2-bf783070fbed/metrics-server/0.log"
Jul 14 01:47:46.203194 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.203168    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-dsczj_307fb8b3-6ab3-44b4-8ac1-3253e552749b/monitoring-plugin/0.log"
Jul 14 01:47:46.240863 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.240820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqlfq_8c0e0edb-e690-47fe-b35c-c8ef1da44b49/node-exporter/0.log"
Jul 14 01:47:46.261101 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.261070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqlfq_8c0e0edb-e690-47fe-b35c-c8ef1da44b49/kube-rbac-proxy/0.log"
Jul 14 01:47:46.283466 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.283440    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jqlfq_8c0e0edb-e690-47fe-b35c-c8ef1da44b49/init-textfile/0.log"
Jul 14 01:47:46.497996 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.497917    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-pn9vp_44919429-543c-4426-9b31-bff4ed86fd96/kube-rbac-proxy-main/0.log"
Jul 14 01:47:46.518304 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.518273    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-pn9vp_44919429-543c-4426-9b31-bff4ed86fd96/kube-rbac-proxy-self/0.log"
Jul 14 01:47:46.538002 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.537977    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-pn9vp_44919429-543c-4426-9b31-bff4ed86fd96/openshift-state-metrics/0.log"
Jul 14 01:47:46.578846 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.578818    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/prometheus/0.log"
Jul 14 01:47:46.603120 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.603063    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/config-reloader/0.log"
Jul 14 01:47:46.623010 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.622981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/thanos-sidecar/0.log"
Jul 14 01:47:46.642942 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.642869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/kube-rbac-proxy-web/0.log"
Jul 14 01:47:46.668396 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.668360    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/kube-rbac-proxy/0.log"
Jul 14 01:47:46.687732 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.687702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/kube-rbac-proxy-thanos/0.log"
Jul 14 01:47:46.710837 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.710806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cb93c8a3-abb6-49aa-9f1c-0c35e9bdd82e/init-config-reloader/0.log"
Jul 14 01:47:46.753758 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.752818    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-hfzz9_c92fd7c8-fb30-40bd-8697-844227edef4f/prometheus-operator/0.log"
Jul 14 01:47:46.768608 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.768583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-hfzz9_c92fd7c8-fb30-40bd-8697-844227edef4f/kube-rbac-proxy/0.log"
Jul 14 01:47:46.916239 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.916206    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4456fc9-rh78x_011dd403-949d-4bea-8f16-c3d6e45ef275/thanos-query/0.log"
Jul 14 01:47:46.944714 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.944481    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4456fc9-rh78x_011dd403-949d-4bea-8f16-c3d6e45ef275/kube-rbac-proxy-web/0.log"
Jul 14 01:47:46.965483 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.965450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4456fc9-rh78x_011dd403-949d-4bea-8f16-c3d6e45ef275/kube-rbac-proxy/0.log"
Jul 14 01:47:46.988052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:46.988009    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4456fc9-rh78x_011dd403-949d-4bea-8f16-c3d6e45ef275/prom-label-proxy/0.log"
Jul 14 01:47:47.009247 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:47.009176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4456fc9-rh78x_011dd403-949d-4bea-8f16-c3d6e45ef275/kube-rbac-proxy-rules/0.log"
Jul 14 01:47:47.064663 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:47.064631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84c4456fc9-rh78x_011dd403-949d-4bea-8f16-c3d6e45ef275/kube-rbac-proxy-metrics/0.log"
Jul 14 01:47:49.660536 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.660437    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"]
Jul 14 01:47:49.668558 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.668531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.671724 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.671674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"]
Jul 14 01:47:49.776977 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.776935    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-lib-modules\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.777161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.777063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-proc\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.777161 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.777138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlqz\" (UniqueName: \"kubernetes.io/projected/218419dc-ace4-4c62-9c42-aa67f0331598-kube-api-access-djlqz\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.777283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.777178    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-sys\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.777283 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.777240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-podres\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878345 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlqz\" (UniqueName: \"kubernetes.io/projected/218419dc-ace4-4c62-9c42-aa67f0331598-kube-api-access-djlqz\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-sys\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-podres\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-lib-modules\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878501    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-sys\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878555 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-proc\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878564    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-proc\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-podres\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.878833 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.878621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/218419dc-ace4-4c62-9c42-aa67f0331598-lib-modules\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.885976 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.885947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlqz\" (UniqueName: \"kubernetes.io/projected/218419dc-ace4-4c62-9c42-aa67f0331598-kube-api-access-djlqz\") pod \"perf-node-gather-daemonset-znzjs\" (UID: \"218419dc-ace4-4c62-9c42-aa67f0331598\") " pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:49.982862 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:49.982828    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:50.157449 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:50.157356    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"]
Jul 14 01:47:51.073810 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.073772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs" event={"ID":"218419dc-ace4-4c62-9c42-aa67f0331598","Type":"ContainerStarted","Data":"794aaa51ebfab38a61c7d2c1a6aa1f87a44ce1806e05d47249907ef87d39a81f"}
Jul 14 01:47:51.074317 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.073870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs" event={"ID":"218419dc-ace4-4c62-9c42-aa67f0331598","Type":"ContainerStarted","Data":"59a8385104606674c196ab459a94c5382f15d1603534b5dcd7f4f4f1309031c3"}
Jul 14 01:47:51.074317 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.073899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:51.092199 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.092136    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs" podStartSLOduration=2.092118109 podStartE2EDuration="2.092118109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 01:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:47:51.089093499 +0000 UTC m=+2872.976358128" watchObservedRunningTime="2026-07-14 01:47:51.092118109 +0000 UTC m=+2872.979382738"
Jul 14 01:47:51.434818 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.434784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zh82v_d9bf370b-db7b-4948-9332-ac45030fbd41/dns/0.log"
Jul 14 01:47:51.456991 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.456944    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zh82v_d9bf370b-db7b-4948-9332-ac45030fbd41/kube-rbac-proxy/0.log"
Jul 14 01:47:51.499098 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:51.499060    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gf759_1b273d22-3594-4e4d-8024-de809ed7af04/dns-node-resolver/0.log"
Jul 14 01:47:52.083593 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:52.083559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z75jz_0150a597-f23b-457a-8f0a-5ded06e898a1/node-ca/0.log"
Jul 14 01:47:53.084287 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:53.084261    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d956c5b-pdkbg_fb370852-4001-4d24-aebe-063138ee8c2c/kube-auth-proxy/0.log"
Jul 14 01:47:53.940019 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:53.939976    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nstzz_4b5a6ad2-1ee6-4761-9705-bcde606f6861/serve-healthcheck-canary/0.log"
Jul 14 01:47:54.474534 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:54.474502    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlcx9_7fec0415-49b6-41d8-a027-8f2ab01240ca/kube-rbac-proxy/0.log"
Jul 14 01:47:54.493648 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:54.493625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlcx9_7fec0415-49b6-41d8-a027-8f2ab01240ca/exporter/0.log"
Jul 14 01:47:54.512777 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:54.512755    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hlcx9_7fec0415-49b6-41d8-a027-8f2ab01240ca/extractor/0.log"
Jul 14 01:47:56.987468 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:56.987437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554b6949c8-6dqbl_721178c4-28cb-40a6-b4dc-45407dafbd9b/manager/0.log"
Jul 14 01:47:57.004081 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:57.004056    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-554b6949c8-6dqbl_721178c4-28cb-40a6-b4dc-45407dafbd9b/copy-manifests/0.log"
Jul 14 01:47:57.091003 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:57.090967    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lrhh6/perf-node-gather-daemonset-znzjs"
Jul 14 01:47:58.284170 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:58.284145    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cb746fd84-4zjm5_ea48cf76-3f3b-4df9-8897-c568f12a8012/manager/0.log"
Jul 14 01:47:59.663976 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:47:59.663944    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d956c5b-pdkbg_fb370852-4001-4d24-aebe-063138ee8c2c/kube-auth-proxy/0.log"
Jul 14 01:48:01.078626 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:01.078541    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54d956c5b-pdkbg_fb370852-4001-4d24-aebe-063138ee8c2c/kube-auth-proxy/0.log"
Jul 14 01:48:04.917096 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:04.917066    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/kube-multus-additional-cni-plugins/0.log"
Jul 14 01:48:04.937703 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:04.937658    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/egress-router-binary-copy/0.log"
Jul 14 01:48:04.956967 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:04.956941    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/cni-plugins/0.log"
Jul 14 01:48:04.975554 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:04.975529    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/bond-cni-plugin/0.log"
Jul 14 01:48:04.995448 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:04.995428    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/routeoverride-cni/0.log"
Jul 14 01:48:05.014052 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:05.014029    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/whereabouts-cni-bincopy/0.log"
Jul 14 01:48:05.033907 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:05.033878    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9w6t8_cbb2e296-cc1c-4cc8-91fb-190d20571b81/whereabouts-cni/0.log"
Jul 14 01:48:05.273550 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:05.273478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w7vdj_e1b7867f-9334-42e0-a0cb-093e277a7f00/kube-multus/0.log"
Jul 14 01:48:05.332097 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:05.332070    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bmkmz_ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b/network-metrics-daemon/0.log"
Jul 14 01:48:05.348743 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:05.348721    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bmkmz_ef7999a0-5121-4f08-9ea0-0b2ed0d7b88b/kube-rbac-proxy/0.log"
Jul 14 01:48:06.206842 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.206811    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-controller/0.log"
Jul 14 01:48:06.222680 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.222648    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/1.log"
Jul 14 01:48:06.222816 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.222710    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovn-acl-logging/0.log"
Jul 14 01:48:06.239720 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.239679    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/kube-rbac-proxy-node/0.log"
Jul 14 01:48:06.258615 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.258588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 14 01:48:06.276472 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.276448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/northd/0.log"
Jul 14 01:48:06.294504 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.294487    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/nbdb/0.log"
Jul 14 01:48:06.313465 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.313446    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/sbdb/0.log"
Jul 14 01:48:06.373348 ip-10-0-138-193 kubenswrapper[2578]: I0714 01:48:06.373325    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-7qzjq_86dc77e7-2874-4b0c-ba5d-5254f189030d/ovnkube-controller/0.log"