Jul 14 17:23:47.967273 ip-10-0-138-98 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 14 17:23:47.967288 ip-10-0-138-98 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 14 17:23:47.967298 ip-10-0-138-98 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 14 17:23:47.967656 ip-10-0-138-98 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 14 17:23:58.061177 ip-10-0-138-98 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 14 17:23:58.061193 ip-10-0-138-98 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 994d9ebaa3f441c6b26f0ace80c84108 --
Jul 14 17:25:53.345629 ip-10-0-138-98 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.710985    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715132    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715145    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715149    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715153    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715156    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715159    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715161    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:25:53.837835 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715164    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:25:53.774544 ip-10-0-138-98 systemd[1]: Started Kubernetes Kubelet.
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715166    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715169    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715172    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715174    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715177    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715179    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715182    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715184    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715187    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715190    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715193    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715196    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715198    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715201    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715203    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715206    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715208    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715211    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715214    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:25:53.839178 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715216    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715218    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715221    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715223    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715226    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715228    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715231    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715234    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715237    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715239    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715242    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715245    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715248    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715250    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715256    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715260    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715263    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715265    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715268    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:25:53.839843 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715276    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715279    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715281    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715284    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715286    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715289    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715291    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715294    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715297    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715299    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715302    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715305    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715307    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715310    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715312    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715315    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715318    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715320    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715323    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715325    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:25:54.013223 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715328    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715330    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715332    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715335    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715337    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715355    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715359    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715361    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715364    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715367    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715370    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715372    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715376    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715379    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715381    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715384    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715387    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715389    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715392    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715395    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:25:54.032018 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715398    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715401    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715403    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715406    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715409    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715412    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715415    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715419    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715424    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715427    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715430    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715433    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.715436    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717718    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717740    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717751    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717838    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717842    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717848    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:25:54.032718 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717853    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717857    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717861    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717865    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717869    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717873    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717877    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717881    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717885    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717889    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717893    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717897    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717901    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717905    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717909    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717913    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717917    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717920    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717925    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717929    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:25:54.033352 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717933    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717937    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717941    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717947    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717951    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717955    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717959    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717963    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717967    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717971    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717974    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717978    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717982    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717986    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717990    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717994    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.717998    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718002    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718006    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718010    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:25:54.034008 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718014    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718016    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718021    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718024    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718026    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718029    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718031    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718034    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718036    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718040    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718043    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718045    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718048    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718050    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718053    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718058    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718062    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718067    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718070    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718074    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:25:54.034815 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718076    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718079    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718082    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718085    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718087    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718090    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718093    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718095    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718098    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718100    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718102    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718105    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718107    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718113    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718116    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718118    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718121    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718133    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718137    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:25:54.035496 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718140    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718143    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718146    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718149    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718152    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718155    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718158    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718160    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718164    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718167    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718170    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718172    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718175    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718177    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718640    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718651    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718654    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718656    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718659    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718663    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:25:54.036163 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718667    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718669    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718672    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718675    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718677    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718680    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718682    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718685    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718688    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718693    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718702    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718705    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718708    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718711    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718713    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718716    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718720    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718724    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718727    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718730    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:25:54.036831 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718733    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718735    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718737    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718742    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718744    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718747    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718749    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718753    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718757    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718760    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718762    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718765    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718768    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718771    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718774    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718777    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718780    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718782    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718785    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:25:54.037500 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718787    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718790    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718794    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718796    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718804    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718807    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718810    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718812    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718815    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718817    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718820    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718822    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718825    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718828    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718830    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718833    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718836    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718838    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718841    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718843    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:25:54.038144 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718846    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718848    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718851    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718853    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718856    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718858    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718860    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718863    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718865    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718868    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718870    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718872    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718875    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718878    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718883    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718886    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718888    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718896    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718899    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718902    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:25:54.038771 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718904    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718907    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718909    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718911    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718914    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718916    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718918    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718921    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718923    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718926    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718928    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718930    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718933    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.718935    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719006    2567 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719014    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719025    2567 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719030    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719033    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719036    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719040    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 17:25:54.039341 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719044    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719047    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719050    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719053    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719056    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719061    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719064    2567 flags.go:64] FLAG: --cgroup-root=""
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719067    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719069    2567 flags.go:64] FLAG: --client-ca-file=""
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719080    2567 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719083    2567 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719090    2567 flags.go:64] FLAG: --cluster-domain=""
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719092    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719095    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719098    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719102    2567 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719106    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719109    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719112    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719115    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719118    2567 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719121    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719124    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719127    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 17:25:54.040047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719131    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719135    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719138    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719141    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719143    2567 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719146    2567 flags.go:64] FLAG: --enable-server="true"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719149    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719155    2567 flags.go:64] FLAG: --event-burst="100"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719159    2567 flags.go:64] FLAG: --event-qps="50"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719162    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719165    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719168    2567 flags.go:64] FLAG: --eviction-hard=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719171    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719175    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719178    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719181    2567 flags.go:64] FLAG: --eviction-soft=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719184    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719187    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719195    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719198    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719201    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719204    2567 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719206    2567 flags.go:64] FLAG: --feature-gates=""
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719210    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719213    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 17:25:54.040716 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719216    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719219    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719222    2567 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719224    2567 flags.go:64] FLAG: --help="false"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719227    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719230    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719233    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719239    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719242    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719245    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719248    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719250    2567 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719253    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719256    2567 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719258    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719261    2567 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719264    2567 flags.go:64] FLAG: --kube-reserved=""
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719267    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719269    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719272    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719276    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719279    2567 flags.go:64] FLAG: --lock-file=""
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719282    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719285    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 17:25:54.041458 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719287    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719293    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719301    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719304    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719307    2567 flags.go:64] FLAG: --logging-format="text"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719310    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719313    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719315    2567 flags.go:64] FLAG: --manifest-url=""
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719318    2567 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719322    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719326    2567 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719329    2567 flags.go:64] FLAG: --max-pods="110"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719332    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719335    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719337    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719355    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719358    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719361    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719364    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719371    2567 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719374    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719377    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719380    2567 flags.go:64] FLAG: --pod-cidr=""
Jul 14 17:25:54.042200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719382    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719387    2567 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719390    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719394    2567 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719397    2567 flags.go:64] FLAG: --port="10250"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719401    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719403    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01c5665f0de4184b0"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719406    2567 flags.go:64] FLAG: --qos-reserved=""
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719409    2567 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719412    2567 flags.go:64] FLAG: --register-node="true"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719415    2567 flags.go:64] FLAG: --register-with-taints=""
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719418    2567 flags.go:64] FLAG: --registry-burst="10"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719427    2567 flags.go:64] FLAG: --registry-qps="5"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719430    2567 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719432    2567 flags.go:64] FLAG: --reserved-memory=""
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719436    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719438    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719441    2567 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719444    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719447    2567 flags.go:64] FLAG: --runonce="false"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719449    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719452    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719455    2567 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719458    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719461    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719466    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 17:25:54.042996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719469    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719471    2567 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719474    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719477    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719479    2567 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719482    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719485    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719488    2567 flags.go:64] FLAG: --system-cgroups=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719491    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719496    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719498    2567 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719502    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719509    2567 flags.go:64] FLAG: --tls-min-version=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719512    2567 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719515    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719518    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719520    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719523    2567 flags.go:64] FLAG: --v="2"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719527    2567 flags.go:64] FLAG: --version="false"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719536    2567 flags.go:64] FLAG: --vmodule=""
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719540    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.719543    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719680    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719684    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:25:54.043870 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719687    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719690    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719693    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719697    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719700    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719703    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719706    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719710    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719713    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719716    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719719    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719722    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719724    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719727    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719729    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719732    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719734    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719737    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719739    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:25:54.044739 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719743    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719745    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719748    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719750    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719753    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719755    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719758    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719760    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719762    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719771    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719773    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719776    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719778    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719781    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719783    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719787    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719790    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719793    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719796    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:25:54.045900 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719798    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719803    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719806    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719808    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719811    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719813    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719815    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719818    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719820    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719823    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719825    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719828    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719830    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719834    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719837    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719839    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719841    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719844    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719847    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:25:54.046705 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719849    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719851    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719854    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719856    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719864    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719867    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719869    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719872    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719874    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719876    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719879    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719881    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719885    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719888    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719893    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719895    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719898    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719900    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719903    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719905    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:25:54.251650 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719907    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719910    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719913    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719915    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719917    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719920    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719923    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719926    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719929    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719931    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719934    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719937    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719939    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719942    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719944    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719947    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719949    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719957    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719960    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:25:54.252677 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.719962    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.720428    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.726533    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.726549    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.726565    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.726571    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726626    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726631    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726634    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726637    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726640    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726643    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726646    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726649    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726651    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:25:54.253275 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726654    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726657    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726659    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726662    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726664    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726667    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726669    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726672    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726675    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726677    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726680    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726682    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726684    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726687    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726689    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726692    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726695    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726698    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726700    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726703    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:25:54.253777 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726706    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726710    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726714    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726717    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726720    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726723    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726726    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726730    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726733    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726736    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726739    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726742    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726744    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726747    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726749    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726753    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726757    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726760    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:25:54.254335 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726762    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726765    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726768    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726770    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726773    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726776    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726779    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726781    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726784    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726786    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726789    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726791    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726794    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726796    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726799    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726802    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726805    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726807    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726810    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726812    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:25:54.254905 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726815    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726817    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726820    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726822    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726824    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726827    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726829    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726832    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726836    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726839    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726841    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726844    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726846    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726852    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726855    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726857    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726860    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726862    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726865    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726867    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:25:54.255477 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726870    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726872    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726874    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726877    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726879    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726882    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726884    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726887    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726890    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726892    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726895    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.726897    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.726902    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727012    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727016    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 17:25:54.256106 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727019    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727022    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727025    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727027    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727030    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727034    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727036    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727039    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727042    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727046    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727048    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727052    2567 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727055    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727057    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727060    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727063    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727066    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727068    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727071    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 17:25:54.256554 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727073    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727076    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727080    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727083    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727086    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727089    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727091    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727096    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727099    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727101    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727103    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727106    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727108    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727111    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727113    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727116    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727118    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727120    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727124    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 17:25:54.257110 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727127    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727129    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727131    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727134    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727137    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727140    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727143    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727145    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727148    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727150    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727152    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727155    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727157    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727160    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727162    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727166    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727169    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727172    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727175    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 17:25:54.257648 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727178    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727180    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727183    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727185    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727188    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727191    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727193    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727196    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727199    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727201    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727203    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727206    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727209    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727211    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727214    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727217    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727219    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727221    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727225    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727227    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 17:25:54.258211 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727230    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727233    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727235    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727238    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727240    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727243    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727245    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727248    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727250    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727253    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727255    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727258    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727260    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727263    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727265    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727268    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727271    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727273    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727276    2567 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 17:25:54.258819 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:53.727278    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.727282    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.727426    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.729205    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.730201    2567 server.go:1019] "Starting client certificate rotation"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.731674    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.731729    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.732638    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.734053    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.744041    2567 log.go:25] "Validated CRI v1 runtime API"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.745757    2567 log.go:25] "Validated CRI v1 image API"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.746419    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 17:25:54.259396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.749815    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9f929738-6933-4dfb-9379-9b3803f1f2f7:/dev/nvme0n1p4 ccca23f3-7a14-49f1-838b-de8ddff3f046:/dev/nvme0n1p3]
Jul 14 17:25:54.259790 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.749828    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.754327    2567 manager.go:216] Machine: {Timestamp:2026-07-14 17:25:53.75323739 +0000 UTC m=+0.297641175 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200104 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec217a1e2167920297f0a046e2d017e9 SystemUUID:ec217a1e-2167-9202-97f0-a046e2d017e9 BootID:994d9eba-a3f4-41c6-b26f-0ace80c84108 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b5:64:eb:79:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b5:64:eb:79:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:59:38:b4:05:d6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.754436    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.754505    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.758020    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.758040    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-98.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.758606    2567 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.758615    2567 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 17:25:54.259905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.758624    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.758706    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.759706    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.760669    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764619    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764731    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764759    2567 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764768    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764783    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764791    2567 kubelet.go:398] "Adding apiserver pod source"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.764799    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.765088    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.765101    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.767181    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768649    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768676    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768833    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768854    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768863    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768870    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768885    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.768891    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.769232    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.769238    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.769244    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.769249    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.769257    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.769264    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.770483    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.770493    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.773655    2567 server.go:1268] "Started kubelet"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.773739    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.773808    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.773896    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.775795    2567 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.777593    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.779173    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.780176    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-98.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.780209    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 17:25:54.260184 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.780207    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.780191    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-98.ec2.internal.18c237fb0c3a7371  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-98.ec2.internal,UID:ip-10-0-138-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-98.ec2.internal,},FirstTimestamp:2026-07-14 17:25:53.773613937 +0000 UTC m=+0.318017721,LastTimestamp:2026-07-14 17:25:53.773613937 +0000 UTC m=+0.318017721,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-98.ec2.internal,}"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.781371    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkfxg"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.782571    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.782573    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.782647    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783420    2567 factory.go:55] Registering systemd factory
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783433    2567 factory.go:223] Registration of the systemd container factory successfully
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783604    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783608    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783623    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783695    2567 factory.go:153] Registering CRI-O factory
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783707    2567 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783729    2567 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783708    2567 factory.go:223] Registration of the crio container factory successfully
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783783    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783810    2567 factory.go:103] Registering Raw factory
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.783823    2567 manager.go:1196] Started watching for new ooms in manager
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.784166    2567 manager.go:349] Starting recovery of all containers
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.787373    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vkfxg"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.788379    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.788427    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-98.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.794561    2567 manager.go:354] Recovery completed
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.799109    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.800926    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.800953    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.800969    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.803021    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.803032    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.803036    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.261311 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.803049    2567 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.803448    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-98.ec2.internal.18c237fb0ddb6978  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-98.ec2.internal,UID:ip-10-0-138-98.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-98.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-98.ec2.internal,},FirstTimestamp:2026-07-14 17:25:53.800939896 +0000 UTC m=+0.345343681,LastTimestamp:2026-07-14 17:25:53.800939896 +0000 UTC m=+0.345343681,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-98.ec2.internal,}"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.805053    2567 policy_none.go:49] "None policy: Start"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.805067    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.805077    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.806657    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.806681    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.806690    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.807050    2567 policy_none.go:47] "Start"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841141    2567 manager.go:337] "Starting Device Plugin manager"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.841180    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841193    2567 server.go:87] "Starting device plugin registration server"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841437    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841565    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841670    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841770    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.841779    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.842225    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.842274    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.942457    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.943366    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.943394    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.943406    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.943433    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:53.951744    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.951765    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-98.ec2.internal\": node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:53.965721    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.043460    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.044873    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.044909    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.044930    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.044964    2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.047355    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:25:54.262175 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.066520    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.149425    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"]
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.149511    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.150442    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.150469    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.150483    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.153545    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.153679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.153718    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.157225    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.157250    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.157261    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.157319    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.157344    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.157357    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.164403    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.164434    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.165126    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.165145    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.165159    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeHasSufficientPID"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.167523    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.172959    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-98.ec2.internal\" not found" node="ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.178979    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-98.ec2.internal\" not found" node="ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.187165    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3df8dd8504f39b6d868d068bcd1a2cff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal\" (UID: \"3df8dd8504f39b6d868d068bcd1a2cff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.187187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df8dd8504f39b6d868d068bcd1a2cff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal\" (UID: \"3df8dd8504f39b6d868d068bcd1a2cff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.263124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.187205    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c5765f711f78dde18a902a1fe60d146-config\") pod \"kube-apiserver-proxy-ip-10-0-138-98.ec2.internal\" (UID: \"8c5765f711f78dde18a902a1fe60d146\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.268177    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.288177    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3df8dd8504f39b6d868d068bcd1a2cff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal\" (UID: \"3df8dd8504f39b6d868d068bcd1a2cff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.288245    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3df8dd8504f39b6d868d068bcd1a2cff-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal\" (UID: \"3df8dd8504f39b6d868d068bcd1a2cff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.288271    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df8dd8504f39b6d868d068bcd1a2cff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal\" (UID: \"3df8dd8504f39b6d868d068bcd1a2cff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.288298    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c5765f711f78dde18a902a1fe60d146-config\") pod \"kube-apiserver-proxy-ip-10-0-138-98.ec2.internal\" (UID: \"8c5765f711f78dde18a902a1fe60d146\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.288336    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3df8dd8504f39b6d868d068bcd1a2cff-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal\" (UID: \"3df8dd8504f39b6d868d068bcd1a2cff\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.288338    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c5765f711f78dde18a902a1fe60d146-config\") pod \"kube-apiserver-proxy-ip-10-0-138-98.ec2.internal\" (UID: \"8c5765f711f78dde18a902a1fe60d146\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.456284 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.368943    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.631759 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.470008    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.631759 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.474445    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.631759 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.481896    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.631759 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.572524    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.773168 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.673338    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-98.ec2.internal\" not found"
Jul 14 17:25:54.773168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.684703    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:25:54.773168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.729905    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 17:25:54.773168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.730077    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 17:25:54.773168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.730077    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 17:25:54.773168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.766997    2567 apiserver.go:52] "Watching apiserver"
Jul 14 17:25:54.784654 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.783569    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 17:25:54.784654 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.783577    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.789623    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.789941    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-zh27r","openshift-network-diagnostics/network-check-target-7knwp","openshift-network-operator/iptables-alerter-k6k67","openshift-ovn-kubernetes/ovnkube-node-pqtvp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5","openshift-cluster-node-tuning-operator/tuned-km94l","openshift-dns/node-resolver-8hg9g","openshift-image-registry/node-ca-qwwj2","openshift-multus/network-metrics-daemon-prqcj","kube-system/global-pull-secret-syncer-69hq7","kube-system/konnectivity-agent-mhnz7","openshift-multus/multus-5rjcc"]
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.792604    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.795006    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.795016    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.795283    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.795314    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.795608    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.795920    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.795925    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.796939    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 17:20:54 +0000 UTC" deadline="2028-04-05 02:11:23.924090244 +0000 UTC"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.796988    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15128h45m29.127105044s"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.797099    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fknmv\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.797136    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.797230    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.797717    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.799534    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.800189    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.800712    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.800772    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.800799    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.800803    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4js4s\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.802751    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.802983    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.802997    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 17:25:54.847596 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.803242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.803276    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.803341    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lll87\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.804479    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.804518    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.805210    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.805407    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.805574    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.805577    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.805981    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lzxkm\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.806649    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.807807    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.807916    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.808016    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.808253    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fs5tj\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.810478    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.810808    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.810820    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bv78g\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.810874    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.813016    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.813058    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.813075    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.813115    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j4hfm\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.813123    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.813152    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.815820    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:54.815881    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.818396    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:54.849871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.819974    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xrldw"
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.821053    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal"]
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.821093    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal"]
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.821166    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.821231    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.821278    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.821828    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vdhb7\""
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.823885    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-l6d42\""
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.824123    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 17:25:54.850763 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.833530    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xrldw"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.884874    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892525    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7j7k\" (UniqueName: \"kubernetes.io/projected/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-kube-api-access-h7j7k\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892556    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b2853cf-fd5f-46a9-94a9-e73045273824-host\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-cnibin\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892589    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-var-lib-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892660    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb46l\" (UniqueName: \"kubernetes.io/projected/3b2853cf-fd5f-46a9-94a9-e73045273824-kube-api-access-fb46l\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892695    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f76fad34-a307-43b9-b220-75c29a93352b-kubelet-config\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-cni-binary-copy\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892750    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-slash\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892770    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-etc-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-socket-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.923148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovnkube-script-lib\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-device-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-kubernetes\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892875    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysctl-conf\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892903    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-daemon-config\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892941    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovnkube-config\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892975    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.892996    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-lib-modules\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893019    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-modprobe-d\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893033    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-sys\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893048    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbfj7\" (UniqueName: \"kubernetes.io/projected/50f0b74c-d249-4659-9b21-0d6f51b22737-kube-api-access-jbfj7\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893068    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-hostroot\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f76fad34-a307-43b9-b220-75c29a93352b-dbus\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893113    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-os-release\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-cni-bin\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893152    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-multus-certs\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893167    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-cnibin\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893190    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-cni-binary-copy\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-kubelet\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893245    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-netns\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893260    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-os-release\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893274    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-systemd\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893295    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-registration-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893315    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-socket-dir-parent\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893331    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jqw\" (UniqueName: \"kubernetes.io/projected/04e6197d-898c-44ef-a067-708a5e78df09-kube-api-access-68jqw\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893346    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-kubelet\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.924698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893381    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-host\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893397    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223f6af3-059f-4613-b60b-787215b37707-tmp-dir\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893411    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-host-slash\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-system-cni-dir\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893471    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-run-netns\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5bg\" (UniqueName: \"kubernetes.io/projected/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-kube-api-access-sd5bg\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893568    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksnb\" (UniqueName: \"kubernetes.io/projected/bd172923-82b5-42c5-affc-c9e5811ab121-kube-api-access-2ksnb\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893592    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f0b74c-d249-4659-9b21-0d6f51b22737-tmp\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893616    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p86j\" (UniqueName: \"kubernetes.io/projected/2277aabd-4d56-4c55-9b26-90214ab70e6d-kube-api-access-4p86j\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893639    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-iptables-alerter-script\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893663    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-log-socket\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893691    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysconfig\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.925712 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893713    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b2853cf-fd5f-46a9-94a9-e73045273824-serviceca\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-cni-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893769    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6ks\" (UniqueName: \"kubernetes.io/projected/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-kube-api-access-xv6ks\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893810    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-ovn\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893872    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-sys-fs\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893902    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-run\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893919    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-k8s-cni-cncf-io\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-etc-kubernetes\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-cni-netd\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.893983    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec9dfb-31b6-4806-92f4-6c36debc59d3-konnectivity-ca\") pod \"konnectivity-agent-mhnz7\" (UID: \"c7ec9dfb-31b6-4806-92f4-6c36debc59d3\") " pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894005    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-conf-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.926220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894029    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-systemd-units\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894044    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-cni-bin\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-env-overrides\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894113    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysctl-d\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894129    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-systemd\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-tuned\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894176    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7ec9dfb-31b6-4806-92f4-6c36debc59d3-agent-certs\") pod \"konnectivity-agent-mhnz7\" (UID: \"c7ec9dfb-31b6-4806-92f4-6c36debc59d3\") " pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894189    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-var-lib-kubelet\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894204    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894226    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-node-log\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-system-cni-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-cni-multus\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894319    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894338    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/223f6af3-059f-4613-b60b-787215b37707-hosts-file\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:54.927239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894356    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlsvr\" (UniqueName: \"kubernetes.io/projected/223f6af3-059f-4613-b60b-787215b37707-kube-api-access-mlsvr\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:54.927733 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:54.894389    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-etc-selinux\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018222    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/223f6af3-059f-4613-b60b-787215b37707-hosts-file\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018269    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlsvr\" (UniqueName: \"kubernetes.io/projected/223f6af3-059f-4613-b60b-787215b37707-kube-api-access-mlsvr\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-etc-selinux\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7j7k\" (UniqueName: \"kubernetes.io/projected/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-kube-api-access-h7j7k\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018351    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/223f6af3-059f-4613-b60b-787215b37707-hosts-file\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018410    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-etc-selinux\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018446    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b2853cf-fd5f-46a9-94a9-e73045273824-host\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018466    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-cnibin\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018483    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-var-lib-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018500    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb46l\" (UniqueName: \"kubernetes.io/projected/3b2853cf-fd5f-46a9-94a9-e73045273824-kube-api-access-fb46l\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f76fad34-a307-43b9-b220-75c29a93352b-kubelet-config\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:55.036775 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-var-lib-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018555    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-cni-binary-copy\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018595    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3b2853cf-fd5f-46a9-94a9-e73045273824-host\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018559    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-cnibin\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018614    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f76fad34-a307-43b9-b220-75c29a93352b-kubelet-config\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018654    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-slash\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.018705    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018724    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-slash\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-etc-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018684    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-etc-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-socket-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.018858    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:25:55.518832623 +0000 UTC m=+2.063236411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:55.037700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018889    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovnkube-script-lib\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018915    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-device-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018940    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-kubernetes\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysctl-conf\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018990    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.018997    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-device-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019016    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-daemon-config\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-kubernetes\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019041    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019086    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysctl-conf\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovnkube-config\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.019155    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019162    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019188    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-lib-modules\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.038830 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.019202    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:25:55.519188772 +0000 UTC m=+2.063592546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-modprobe-d\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019242    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-sys\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbfj7\" (UniqueName: \"kubernetes.io/projected/50f0b74c-d249-4659-9b21-0d6f51b22737-kube-api-access-jbfj7\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-hostroot\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019291    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-lib-modules\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019294    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f76fad34-a307-43b9-b220-75c29a93352b-dbus\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019329    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-os-release\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019347    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-cni-bin\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-multus-certs\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019381    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-cnibin\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019404    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-cni-binary-copy\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019436    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f76fad34-a307-43b9-b220-75c29a93352b-dbus\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-modprobe-d\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019546    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-sys\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.039532 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019642    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-socket-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019684    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-cni-bin\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-kubelet\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-hostroot\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019711    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-netns\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-netns\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019746    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-os-release\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-cnibin\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019811    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-systemd\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019830    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-os-release\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019862    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-os-release\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021162    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021184    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.019787    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-systemd\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.040218 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021277    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-multus-certs\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021285    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-registration-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-socket-dir-parent\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021341    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-kubelet\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021345    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jqw\" (UniqueName: \"kubernetes.io/projected/04e6197d-898c-44ef-a067-708a5e78df09-kube-api-access-68jqw\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-kubelet\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021406    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-host\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223f6af3-059f-4613-b60b-787215b37707-tmp-dir\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021471    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-host-slash\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-cni-binary-copy\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-daemon-config\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-host\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-system-cni-dir\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021572    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-host-slash\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.040909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021573    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021613    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-run-netns\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021622    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovnkube-config\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021633    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-socket-dir-parent\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021639    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5bg\" (UniqueName: \"kubernetes.io/projected/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-kube-api-access-sd5bg\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksnb\" (UniqueName: \"kubernetes.io/projected/bd172923-82b5-42c5-affc-c9e5811ab121-kube-api-access-2ksnb\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-run-netns\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021695    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021702    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f0b74c-d249-4659-9b21-0d6f51b22737-tmp\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021705    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04e6197d-898c-44ef-a067-708a5e78df09-cni-binary-copy\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-kubelet\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p86j\" (UniqueName: \"kubernetes.io/projected/2277aabd-4d56-4c55-9b26-90214ab70e6d-kube-api-access-4p86j\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021765    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04e6197d-898c-44ef-a067-708a5e78df09-system-cni-dir\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021775    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-registration-dir\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.041590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-iptables-alerter-script\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021837    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-log-socket\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021852    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/223f6af3-059f-4613-b60b-787215b37707-tmp-dir\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysconfig\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b2853cf-fd5f-46a9-94a9-e73045273824-serviceca\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021928    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-cni-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021947    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-log-socket\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6ks\" (UniqueName: \"kubernetes.io/projected/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-kube-api-access-xv6ks\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.021981    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-ovn\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022036    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysconfig\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022060    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-sys-fs\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022010    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-cni-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022084    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-run\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.044353 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022107    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-ovn\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-k8s-cni-cncf-io\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-run-k8s-cni-cncf-io\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022144    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-etc-kubernetes\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-etc-kubernetes\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022177    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovnkube-script-lib\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022186    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-cni-netd\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-run\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bd172923-82b5-42c5-affc-c9e5811ab121-sys-fs\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022241    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-run-openvswitch\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022241    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec9dfb-31b6-4806-92f4-6c36debc59d3-konnectivity-ca\") pod \"konnectivity-agent-mhnz7\" (UID: \"c7ec9dfb-31b6-4806-92f4-6c36debc59d3\") " pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022263    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-run-ovn-kubernetes\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022993    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3b2853cf-fd5f-46a9-94a9-e73045273824-serviceca\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.045060 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022285    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-cni-netd\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023391    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-conf-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.022336    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-iptables-alerter-script\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-multus-conf-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-systemd-units\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023641    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-cni-bin\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-env-overrides\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysctl-d\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-systemd\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023836    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-tuned\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023872    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7ec9dfb-31b6-4806-92f4-6c36debc59d3-agent-certs\") pod \"konnectivity-agent-mhnz7\" (UID: \"c7ec9dfb-31b6-4806-92f4-6c36debc59d3\") " pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023905    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-var-lib-kubelet\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023967    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-node-log\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.023999    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-system-cni-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.045820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-cni-multus\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024089    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7ec9dfb-31b6-4806-92f4-6c36debc59d3-konnectivity-ca\") pod \"konnectivity-agent-mhnz7\" (UID: \"c7ec9dfb-31b6-4806-92f4-6c36debc59d3\") " pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024110    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-env-overrides\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024137    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-host-var-lib-cni-multus\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024197    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-systemd-units\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-host-cni-bin\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024304    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-var-lib-kubelet\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024543    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-node-log\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024632    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-system-cni-dir\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024669    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-systemd\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.024805    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-sysctl-d\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.025016    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f0b74c-d249-4659-9b21-0d6f51b22737-tmp\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.025145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-ovn-node-metrics-cert\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.027059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50f0b74c-d249-4659-9b21-0d6f51b22737-etc-tuned\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.027243    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7ec9dfb-31b6-4806-92f4-6c36debc59d3-agent-certs\") pod \"konnectivity-agent-mhnz7\" (UID: \"c7ec9dfb-31b6-4806-92f4-6c36debc59d3\") " pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:55.046866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.028576    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7j7k\" (UniqueName: \"kubernetes.io/projected/e5f6b17b-9a7e-4cf5-b92f-07f01c261171-kube-api-access-h7j7k\") pod \"iptables-alerter-k6k67\" (UID: \"e5f6b17b-9a7e-4cf5-b92f-07f01c261171\") " pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.048555 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.048258    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jqw\" (UniqueName: \"kubernetes.io/projected/04e6197d-898c-44ef-a067-708a5e78df09-kube-api-access-68jqw\") pod \"multus-additional-cni-plugins-zh27r\" (UID: \"04e6197d-898c-44ef-a067-708a5e78df09\") " pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.048555 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.048326    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb46l\" (UniqueName: \"kubernetes.io/projected/3b2853cf-fd5f-46a9-94a9-e73045273824-kube-api-access-fb46l\") pod \"node-ca-qwwj2\" (UID: \"3b2853cf-fd5f-46a9-94a9-e73045273824\") " pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.048555 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.048339    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbfj7\" (UniqueName: \"kubernetes.io/projected/50f0b74c-d249-4659-9b21-0d6f51b22737-kube-api-access-jbfj7\") pod \"tuned-km94l\" (UID: \"50f0b74c-d249-4659-9b21-0d6f51b22737\") " pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.048555 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.048445    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6ks\" (UniqueName: \"kubernetes.io/projected/acbf3934-a6a1-49a6-b56e-edcd941ee2d8-kube-api-access-xv6ks\") pod \"multus-5rjcc\" (UID: \"acbf3934-a6a1-49a6-b56e-edcd941ee2d8\") " pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.048940 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.048760    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:25:55.048940 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.048796    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:25:55.048940 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.048819    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:55.048940 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.048900    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:25:55.548874049 +0000 UTC m=+2.093277846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:55.049404 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.049138    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksnb\" (UniqueName: \"kubernetes.io/projected/bd172923-82b5-42c5-affc-c9e5811ab121-kube-api-access-2ksnb\") pod \"aws-ebs-csi-driver-node-gzmd5\" (UID: \"bd172923-82b5-42c5-affc-c9e5811ab121\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.050500    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlsvr\" (UniqueName: \"kubernetes.io/projected/223f6af3-059f-4613-b60b-787215b37707-kube-api-access-mlsvr\") pod \"node-resolver-8hg9g\" (UID: \"223f6af3-059f-4613-b60b-787215b37707\") " pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.050579    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p86j\" (UniqueName: \"kubernetes.io/projected/2277aabd-4d56-4c55-9b26-90214ab70e6d-kube-api-access-4p86j\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.051185    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5bg\" (UniqueName: \"kubernetes.io/projected/cf8d762e-4ca2-4835-a4c0-e4eba74ad85f-kube-api-access-sd5bg\") pod \"ovnkube-node-pqtvp\" (UID: \"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f\") " pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.057666    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.061739    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.082951    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-km94l"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.098590    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8hg9g"
Jul 14 17:25:55.138445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.135760    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qwwj2"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.152697    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.156146    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.161417    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5rjcc"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.228403    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5765f711f78dde18a902a1fe60d146.slice/crio-e3efb5dd7d7195ed285a9fba2854159870270777c0d204cfbc27ed995718fed1 WatchSource:0}: Error finding container e3efb5dd7d7195ed285a9fba2854159870270777c0d204cfbc27ed995718fed1: Status 404 returned error can't find the container with id e3efb5dd7d7195ed285a9fba2854159870270777c0d204cfbc27ed995718fed1
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.228890    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df8dd8504f39b6d868d068bcd1a2cff.slice/crio-56c2b820f0060b6503381d535473df3dc5bba4e4b64bd28a86602750c8f94636 WatchSource:0}: Error finding container 56c2b820f0060b6503381d535473df3dc5bba4e4b64bd28a86602750c8f94636: Status 404 returned error can't find the container with id 56c2b820f0060b6503381d535473df3dc5bba4e4b64bd28a86602750c8f94636
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.232237    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.274770    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zh27r"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.284081    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e6197d_898c_44ef_a067_708a5e78df09.slice/crio-d5f75e53866f9530c52bdde9e740b92e24a3e4a074d9365eb317e21002e4a871 WatchSource:0}: Error finding container d5f75e53866f9530c52bdde9e740b92e24a3e4a074d9365eb317e21002e4a871: Status 404 returned error can't find the container with id d5f75e53866f9530c52bdde9e740b92e24a3e4a074d9365eb317e21002e4a871
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.288043    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k6k67"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.295054    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:25:55.375103 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.298012    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f6b17b_9a7e_4cf5_b92f_07f01c261171.slice/crio-78c6d60c7773909d2b66db0ca6ed194d3ddca120d385a1c7148732e92b2d16ff WatchSource:0}: Error finding container 78c6d60c7773909d2b66db0ca6ed194d3ddca120d385a1c7148732e92b2d16ff: Status 404 returned error can't find the container with id 78c6d60c7773909d2b66db0ca6ed194d3ddca120d385a1c7148732e92b2d16ff
Jul 14 17:25:55.510065 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.472078    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223f6af3_059f_4613_b60b_787215b37707.slice/crio-62e3de1d9be7fc5d385f8bde0dcb5392d1d94a2f7c0f30c4be7b5dea5a51eba5 WatchSource:0}: Error finding container 62e3de1d9be7fc5d385f8bde0dcb5392d1d94a2f7c0f30c4be7b5dea5a51eba5: Status 404 returned error can't find the container with id 62e3de1d9be7fc5d385f8bde0dcb5392d1d94a2f7c0f30c4be7b5dea5a51eba5
Jul 14 17:25:55.605703 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.527983    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:55.605703 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.528024    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:55.605703 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.528156    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:55.605703 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.528214    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:25:56.528197866 +0000 UTC m=+3.072601642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:55.605703 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.528156    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:55.605703 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.528298    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:25:56.528283331 +0000 UTC m=+3.072687104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.613249    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8d762e_4ca2_4835_a4c0_e4eba74ad85f.slice/crio-eb695f35e857e48c55ad4d707ca0545e0f31c367d01eb85d8df2694b3c17c9ba WatchSource:0}: Error finding container eb695f35e857e48c55ad4d707ca0545e0f31c367d01eb85d8df2694b3c17c9ba: Status 404 returned error can't find the container with id eb695f35e857e48c55ad4d707ca0545e0f31c367d01eb85d8df2694b3c17c9ba
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.614309    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f0b74c_d249_4659_9b21_0d6f51b22737.slice/crio-41140c77eaff852ba4d69a238e053d0b22821b2228ca12fa030c8a6a621ad654 WatchSource:0}: Error finding container 41140c77eaff852ba4d69a238e053d0b22821b2228ca12fa030c8a6a621ad654: Status 404 returned error can't find the container with id 41140c77eaff852ba4d69a238e053d0b22821b2228ca12fa030c8a6a621ad654
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.614925    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2853cf_fd5f_46a9_94a9_e73045273824.slice/crio-43080d0970c2699d08c724c3c474448ef217d69f4db0e9e08a17187e80b7c0be WatchSource:0}: Error finding container 43080d0970c2699d08c724c3c474448ef217d69f4db0e9e08a17187e80b7c0be: Status 404 returned error can't find the container with id 43080d0970c2699d08c724c3c474448ef217d69f4db0e9e08a17187e80b7c0be
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.619858    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd172923_82b5_42c5_affc_c9e5811ab121.slice/crio-5285754169ce3a0b87bba18b8d965f4ba017a7e21d3723b5c2ad9ae39304d943 WatchSource:0}: Error finding container 5285754169ce3a0b87bba18b8d965f4ba017a7e21d3723b5c2ad9ae39304d943: Status 404 returned error can't find the container with id 5285754169ce3a0b87bba18b8d965f4ba017a7e21d3723b5c2ad9ae39304d943
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.631308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.631474    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.631489    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.631498    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:55.631542    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:25:56.631527377 +0000 UTC m=+3.175931151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:55.746227 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.647404    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbf3934_a6a1_49a6_b56e_edcd941ee2d8.slice/crio-c3bbe52935598e61d9c59cb5b3dc248e4d353a08326cff2cabee0a4ab8d648c9 WatchSource:0}: Error finding container c3bbe52935598e61d9c59cb5b3dc248e4d353a08326cff2cabee0a4ab8d648c9: Status 404 returned error can't find the container with id c3bbe52935598e61d9c59cb5b3dc248e4d353a08326cff2cabee0a4ab8d648c9
Jul 14 17:25:55.748811 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:25:55.747827    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ec9dfb_31b6_4806_92f4_6c36debc59d3.slice/crio-faeae7d7aa939aceff03489a1369d76ac716e3557b8246990ec69f89bbb3bfd2 WatchSource:0}: Error finding container faeae7d7aa939aceff03489a1369d76ac716e3557b8246990ec69f89bbb3bfd2: Status 404 returned error can't find the container with id faeae7d7aa939aceff03489a1369d76ac716e3557b8246990ec69f89bbb3bfd2
Jul 14 17:25:55.879632 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.836379    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 17:20:55 +0000 UTC" deadline="2028-01-14 18:59:13.194629738 +0000 UTC"
Jul 14 17:25:55.879632 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:55.836437    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13177h33m17.358196192s"
Jul 14 17:25:56.048603 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.048570    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:56.048766 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.048703    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:25:56.049129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.049102    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:56.049237 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.049201    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:25:56.055019 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.054933    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rjcc" event={"ID":"acbf3934-a6a1-49a6-b56e-edcd941ee2d8","Type":"ContainerStarted","Data":"c3bbe52935598e61d9c59cb5b3dc248e4d353a08326cff2cabee0a4ab8d648c9"}
Jul 14 17:25:56.059253 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.059172    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5" event={"ID":"bd172923-82b5-42c5-affc-c9e5811ab121","Type":"ContainerStarted","Data":"5285754169ce3a0b87bba18b8d965f4ba017a7e21d3723b5c2ad9ae39304d943"}
Jul 14 17:25:56.061154 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.061052    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"eb695f35e857e48c55ad4d707ca0545e0f31c367d01eb85d8df2694b3c17c9ba"}
Jul 14 17:25:56.064343 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.064258    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-km94l" event={"ID":"50f0b74c-d249-4659-9b21-0d6f51b22737","Type":"ContainerStarted","Data":"41140c77eaff852ba4d69a238e053d0b22821b2228ca12fa030c8a6a621ad654"}
Jul 14 17:25:56.067358 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.067285    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwwj2" event={"ID":"3b2853cf-fd5f-46a9-94a9-e73045273824","Type":"ContainerStarted","Data":"43080d0970c2699d08c724c3c474448ef217d69f4db0e9e08a17187e80b7c0be"}
Jul 14 17:25:56.069900 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.069874    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hg9g" event={"ID":"223f6af3-059f-4613-b60b-787215b37707","Type":"ContainerStarted","Data":"62e3de1d9be7fc5d385f8bde0dcb5392d1d94a2f7c0f30c4be7b5dea5a51eba5"}
Jul 14 17:25:56.073895 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.073828    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6k67" event={"ID":"e5f6b17b-9a7e-4cf5-b92f-07f01c261171","Type":"ContainerStarted","Data":"78c6d60c7773909d2b66db0ca6ed194d3ddca120d385a1c7148732e92b2d16ff"}
Jul 14 17:25:56.076317 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.076248    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerStarted","Data":"d5f75e53866f9530c52bdde9e740b92e24a3e4a074d9365eb317e21002e4a871"}
Jul 14 17:25:56.077710 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.077644    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal" event={"ID":"8c5765f711f78dde18a902a1fe60d146","Type":"ContainerStarted","Data":"e3efb5dd7d7195ed285a9fba2854159870270777c0d204cfbc27ed995718fed1"}
Jul 14 17:25:56.082391 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.082143    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal" event={"ID":"3df8dd8504f39b6d868d068bcd1a2cff","Type":"ContainerStarted","Data":"56c2b820f0060b6503381d535473df3dc5bba4e4b64bd28a86602750c8f94636"}
Jul 14 17:25:56.084445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.084396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mhnz7" event={"ID":"c7ec9dfb-31b6-4806-92f4-6c36debc59d3","Type":"ContainerStarted","Data":"faeae7d7aa939aceff03489a1369d76ac716e3557b8246990ec69f89bbb3bfd2"}
Jul 14 17:25:56.247453 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.247403    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:25:56.540623 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.540020    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:56.540623 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.540116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:56.540623 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.540267    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:56.540623 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.540331    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:25:58.540310965 +0000 UTC m=+5.084714742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:56.540623 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.540542    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:56.540623 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.540597    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:25:58.540582509 +0000 UTC m=+5.084986289 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:56.640753 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.640694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:56.641056 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.640873    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:25:56.641056 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.640899    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:25:56.641056 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.640914    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:56.641056 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:56.640981    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:25:58.640957089 +0000 UTC m=+5.185360880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:56.837469 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.837291    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 17:20:55 +0000 UTC" deadline="2028-02-13 21:21:09.342328828 +0000 UTC"
Jul 14 17:25:56.837469 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:56.837368    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13899h55m12.50496523s"
Jul 14 17:25:57.046104 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:57.045557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:57.046104 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:57.045696    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:25:57.781850 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:57.781819    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 17:25:58.045971 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:58.045939    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:58.046162 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.046085    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:25:58.046162 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:58.046141    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:58.046609 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.046267    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:25:58.558830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:58.558033    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:58.558830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:58.558142    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:25:58.558830 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.558276    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:58.558830 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.558346    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:26:02.55832637 +0000 UTC m=+9.102730147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:25:58.558830 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.558434    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:58.558830 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.558472    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:26:02.558458757 +0000 UTC m=+9.102862536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:25:58.660117 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:58.659292    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:25:58.660117 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.659532    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:25:58.660117 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.659554    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:25:58.660117 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.659567    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:58.660117 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:58.659627    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:26:02.659608149 +0000 UTC m=+9.204011926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:25:59.045677 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:25:59.045593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:25:59.045843 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:25:59.045761    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:00.046523 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:00.046487    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:00.047000 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:00.046618    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:00.048617 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:00.048431    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:00.048617 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:00.048558    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:01.046080 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:01.045550    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:01.046080 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:01.045694    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:02.046249 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:02.046156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:02.046697 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.046309    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:02.046756 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:02.046162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:02.046832 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.046804    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:02.596168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:02.595830    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:02.596168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:02.595885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:02.596168 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.596006    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:26:02.596168 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.596042    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:26:02.596168 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.596084    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:26:10.596062949 +0000 UTC m=+17.140466739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:26:02.596168 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.596139    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:26:10.596120207 +0000 UTC m=+17.140523998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:26:02.696798 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:02.696704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:02.697016 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.696904    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:26:02.697016 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.696928    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:26:02.697016 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.696939    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:26:02.697016 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:02.696999    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:26:10.6969779 +0000 UTC m=+17.241381676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:26:03.045772 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:03.045686    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:03.045937 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:03.045846    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:04.046631 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:04.046488    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:04.046631 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:04.046541    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:04.047164 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:04.046662    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:04.047164 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:04.046800    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:05.045916 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:05.045881    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:05.046099 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:05.046039    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:06.046012 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:06.045979    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:06.046454 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:06.046135    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:06.046454 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:06.046190    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:06.046454 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:06.046292    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:07.045528 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:07.045492    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:07.045700 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:07.045600    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:08.045860 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:08.045828    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:08.046262 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:08.045950    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:08.046262 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:08.046016    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:08.046262 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:08.046149    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:09.045976 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:09.045934    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:09.046469 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:09.046075    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:10.045737 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:10.045688    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:10.045913 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.045827    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:10.045913 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:10.045867    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:10.046038 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.045936    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:10.661754 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:10.661713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:10.661754 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:10.661767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:10.661982 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.661875    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:26:10.661982 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.661952    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:26:26.661929681 +0000 UTC m=+33.206333455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:26:10.661982 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.661882    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:26:10.662114 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.662026    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:26:26.662008411 +0000 UTC m=+33.206412188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:26:10.762119 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:10.762075    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:10.762275 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.762222    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:26:10.762275 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.762241    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:26:10.762275 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.762252    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:26:10.762460 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:10.762313    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:26:26.762295413 +0000 UTC m=+33.306699202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:26:11.045566 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:11.045489    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:11.045706 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:11.045616    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:12.045728 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:12.045692    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:12.045728 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:12.045729    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:12.046239 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:12.045817    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:12.046239 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:12.045947    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:13.045509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:13.045474    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:13.045668 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:13.045608    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:14.045812 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:14.045779    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:14.046261 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:14.045884    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:14.046261 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:14.045935    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:14.046261 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:14.046048    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:15.046182 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:15.046142    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:15.046593 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:15.046280    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:16.045290 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.045266    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:16.045429 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:16.045410    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:16.045499 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.045455    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:16.045570 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:16.045552    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:16.139134 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.139101    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5rjcc" event={"ID":"acbf3934-a6a1-49a6-b56e-edcd941ee2d8","Type":"ContainerStarted","Data":"0868105a903a28a9cc85d9b79eb66c1c5cfa0cddf466dd2dde84bada5a08167a"}
Jul 14 17:26:16.141331 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.141307    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-km94l" event={"ID":"50f0b74c-d249-4659-9b21-0d6f51b22737","Type":"ContainerStarted","Data":"755e9726fa0dc31d303830c5ee3cabcceedf6233544766a42c0f2c61374ac090"}
Jul 14 17:26:16.142793 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.142684    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal" event={"ID":"8c5765f711f78dde18a902a1fe60d146","Type":"ContainerStarted","Data":"81371f17b3d8b2c00635e7330af83c823633614c9bfe305d1826398d32e3caf3"}
Jul 14 17:26:16.155186 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.155093    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5rjcc" podStartSLOduration=1.9032374810000001 podStartE2EDuration="22.155082238s" totalImagesPullingTime="20.251844757s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:16.15464763 +0000 UTC m=+22.699051424" watchObservedRunningTime="2026-07-14 17:26:16.155082238 +0000 UTC m=+22.699486032"
Jul 14 17:26:16.166973 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.166940    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-98.ec2.internal" podStartSLOduration=21.166927338 podStartE2EDuration="21.166927338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:16.166603566 +0000 UTC m=+22.711007361" watchObservedRunningTime="2026-07-14 17:26:16.166927338 +0000 UTC m=+22.711331208"
Jul 14 17:26:16.182873 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:16.182827    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-km94l" podStartSLOduration=1.9307584979999999 podStartE2EDuration="22.182809137s" totalImagesPullingTime="20.252050639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:16.182420297 +0000 UTC m=+22.726824090" watchObservedRunningTime="2026-07-14 17:26:16.182809137 +0000 UTC m=+22.727212932"
Jul 14 17:26:17.046225 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.045943    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:17.046336 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:17.046309    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:17.149044 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.149001    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mhnz7" event={"ID":"c7ec9dfb-31b6-4806-92f4-6c36debc59d3","Type":"ContainerStarted","Data":"366aa10b4c7b672f2bc53787eddb669bd8cd6be4aebb7090618a9864077d276c"}
Jul 14 17:26:17.150660 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.150636    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5" event={"ID":"bd172923-82b5-42c5-affc-c9e5811ab121","Type":"ContainerStarted","Data":"2eec114708dccf39213a928cd8609931c4e9d4ab02f3d4d4ab51f7abaf22f9aa"}
Jul 14 17:26:17.153324 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153306    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:26:17.153630 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153607    2567 generic.go:356] "Generic (PLEG): container finished" podID="cf8d762e-4ca2-4835-a4c0-e4eba74ad85f" containerID="4a0b1df0d758efb6ff8cfd49972a6af1a945b2a7fa8f7fa4d5150231fddaad12" exitCode=1
Jul 14 17:26:17.153724 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153663    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"c9be3be0b5ff7a446cd8de01b9d9d1fb7a684b570e01ba98aa1b2cee834ececb"}
Jul 14 17:26:17.153724 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153717    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"ee7deec98be5c51f6843f6cdf6f039d65c04a88244547ca494c3a20b9124a1ce"}
Jul 14 17:26:17.153830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153761    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"adf9e882bc408b64cc708ca62384f12d9fe12b8401a35063e5d98787eaf7a45d"}
Jul 14 17:26:17.153830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"2cab0cb35045d059f471af11ec3cf1cb79ae75a3cbff3d379a1cc27c487d6183"}
Jul 14 17:26:17.153830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153820    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerDied","Data":"4a0b1df0d758efb6ff8cfd49972a6af1a945b2a7fa8f7fa4d5150231fddaad12"}
Jul 14 17:26:17.153927 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.153852    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"1c65d4f1c2776c9db00dcc95c1f3d1172a2b2ff8cdec2fbf1cdcfd7676e879cc"}
Jul 14 17:26:17.154908 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.154887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qwwj2" event={"ID":"3b2853cf-fd5f-46a9-94a9-e73045273824","Type":"ContainerStarted","Data":"1acb0bea65d603ad05d2856743d8500fb7555a261996ff41e9d5ebcccca711ce"}
Jul 14 17:26:17.156221 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.156202    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8hg9g" event={"ID":"223f6af3-059f-4613-b60b-787215b37707","Type":"ContainerStarted","Data":"389a7a9ef4eb4881e9ed353104ac597e05ad3f6fe71185ad9bcb0cfc2b021d0a"}
Jul 14 17:26:17.157509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.157489    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k6k67" event={"ID":"e5f6b17b-9a7e-4cf5-b92f-07f01c261171","Type":"ContainerStarted","Data":"b57879e210423412e30a7d9432f78162e66fbb7781e70d1e2acbd2a905f5dd8d"}
Jul 14 17:26:17.158842 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.158823    2567 generic.go:356] "Generic (PLEG): container finished" podID="04e6197d-898c-44ef-a067-708a5e78df09" containerID="f65da91233656b5e1406526fb19e10d5789e29df9674d04feec2e1432c0b8eb7" exitCode=0
Jul 14 17:26:17.158904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.158880    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerDied","Data":"f65da91233656b5e1406526fb19e10d5789e29df9674d04feec2e1432c0b8eb7"}
Jul 14 17:26:17.160313 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.160277    2567 generic.go:356] "Generic (PLEG): container finished" podID="3df8dd8504f39b6d868d068bcd1a2cff" containerID="f048d4575872d957c6c0b81ba6a7b03e7e697224e667f76c1a27338698fa724c" exitCode=0
Jul 14 17:26:17.160409 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.160363    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal" event={"ID":"3df8dd8504f39b6d868d068bcd1a2cff","Type":"ContainerDied","Data":"f048d4575872d957c6c0b81ba6a7b03e7e697224e667f76c1a27338698fa724c"}
Jul 14 17:26:17.163865 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.163832    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mhnz7" podStartSLOduration=3.045440032 podStartE2EDuration="23.163821623s" totalImagesPullingTime="20.118381591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:17.163469781 +0000 UTC m=+23.707873573" watchObservedRunningTime="2026-07-14 17:26:17.163821623 +0000 UTC m=+23.708225416"
Jul 14 17:26:17.213180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.213091    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k6k67" podStartSLOduration=2.645931201 podStartE2EDuration="23.213071259s" totalImagesPullingTime="20.567140058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:17.212420405 +0000 UTC m=+23.756824199" watchObservedRunningTime="2026-07-14 17:26:17.213071259 +0000 UTC m=+23.757475050"
Jul 14 17:26:17.246059 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.246008    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8hg9g" podStartSLOduration=2.854810552 podStartE2EDuration="23.245989392s" totalImagesPullingTime="20.39117884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:17.228291902 +0000 UTC m=+23.772695697" watchObservedRunningTime="2026-07-14 17:26:17.245989392 +0000 UTC m=+23.790393186"
Jul 14 17:26:17.246167 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.246092    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qwwj2" podStartSLOduration=3.018710107 podStartE2EDuration="23.246087153s" totalImagesPullingTime="20.227377046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:17.245813572 +0000 UTC m=+23.790217365" watchObservedRunningTime="2026-07-14 17:26:17.246087153 +0000 UTC m=+23.790490946"
Jul 14 17:26:17.980671 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:17.980648    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 17:26:18.045722 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.045694    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:18.045889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.045694    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:18.045889 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:18.045816    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:18.046004 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:18.045912    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:18.165465 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.165378    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal" event={"ID":"3df8dd8504f39b6d868d068bcd1a2cff","Type":"ContainerStarted","Data":"75ea0cc9a4e90cbd3d11e821b957ff864bb9391a8ddafc5ea820d08245bc5a19"}
Jul 14 17:26:18.167429 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.167396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5" event={"ID":"bd172923-82b5-42c5-affc-c9e5811ab121","Type":"ContainerStarted","Data":"213f183636131bf03945750dcf5b355c6ea952586a17971e71cd8efe65a124a3"}
Jul 14 17:26:18.184842 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.184801    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-98.ec2.internal" podStartSLOduration=23.184786618 podStartE2EDuration="23.184786618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:18.184154645 +0000 UTC m=+24.728558439" watchObservedRunningTime="2026-07-14 17:26:18.184786618 +0000 UTC m=+24.729190411"
Jul 14 17:26:18.855085 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.854978    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T17:26:17.98066559Z","UUID":"4f944246-cbf3-4ee4-b5d7-30c630371541","Handler":null,"Name":"","Endpoint":""}
Jul 14 17:26:18.856729 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.856707    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 17:26:18.856853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:18.856744    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 17:26:19.045818 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:19.045788    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:19.045960 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:19.045902    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:20.046312 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.046113    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:20.046831 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.046175    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:20.046831 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:20.046429    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:20.046831 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:20.046567    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:20.174036 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.173949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5" event={"ID":"bd172923-82b5-42c5-affc-c9e5811ab121","Type":"ContainerStarted","Data":"4d8578bf61bf2cd92cb534b2033acf01489796f21c945d8893b21048bad81584"}
Jul 14 17:26:20.177146 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.177098    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:26:20.177516 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.177493    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"1ee70a7b86afacbcd9036024bb7f0bb60e139d806e04f2b11039c7a3b719df23"}
Jul 14 17:26:20.192708 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.192666    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gzmd5" podStartSLOduration=2.910864198 podStartE2EDuration="26.192653921s" totalImagesPullingTime="23.281789723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:20.192314405 +0000 UTC m=+26.736718199" watchObservedRunningTime="2026-07-14 17:26:20.192653921 +0000 UTC m=+26.737057714"
Jul 14 17:26:20.222435 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.222411    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:26:20.223149 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:20.223131    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:26:21.045191 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:21.045157    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:21.045401 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:21.045302    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:21.180500 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:21.180467    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:26:21.181037 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:21.181019    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mhnz7"
Jul 14 17:26:22.045472 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.045236    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:22.045631 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.045313    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:22.045631 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:22.045497    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:22.045631 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:22.045571    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:22.185395 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.185369    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:26:22.185981 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.185745    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"b1bfda250f357ef4b4680bf6ccb9c162f6786772884ada12a7e9eee7f76f7cd6"}
Jul 14 17:26:22.186041 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.185978    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:26:22.186236 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.186221    2567 scope.go:117] "RemoveContainer" containerID="4a0b1df0d758efb6ff8cfd49972a6af1a945b2a7fa8f7fa4d5150231fddaad12"
Jul 14 17:26:22.187606 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.187584    2567 generic.go:356] "Generic (PLEG): container finished" podID="04e6197d-898c-44ef-a067-708a5e78df09" containerID="57fb26a83fab1133d7d9d8bdf41f64ef2efd53bcb84120d70326824a60d1add2" exitCode=0
Jul 14 17:26:22.187731 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.187671    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerDied","Data":"57fb26a83fab1133d7d9d8bdf41f64ef2efd53bcb84120d70326824a60d1add2"}
Jul 14 17:26:22.201677 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.201658    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:26:22.808367 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:22.807814    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hg9g_223f6af3-059f-4613-b60b-787215b37707/dns-node-resolver/0.log"
Jul 14 17:26:23.045384 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.045329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:23.045636 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:23.045504    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:23.193693 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.193623    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:26:23.194266 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.193942    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" event={"ID":"cf8d762e-4ca2-4835-a4c0-e4eba74ad85f","Type":"ContainerStarted","Data":"569b25f5d8f843dfe7efe88c20f72866cd5c564f9ff4fbb4736cfd08e5b6c71b"}
Jul 14 17:26:23.194374 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.194321    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:26:23.194437 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.194402    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:26:23.209082 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.209062    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:26:23.225472 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.225430    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp" podStartSLOduration=8.56138544 podStartE2EDuration="29.225416967s" totalImagesPullingTime="20.664031527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:23.224405175 +0000 UTC m=+29.768808969" watchObservedRunningTime="2026-07-14 17:26:23.225416967 +0000 UTC m=+29.769820761"
Jul 14 17:26:23.492470 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.491189    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prqcj"]
Jul 14 17:26:23.492470 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.491589    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:23.492470 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:23.491770    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:23.494082 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.494055    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7knwp"]
Jul 14 17:26:23.494176 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.494162    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:23.494279 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:23.494259    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:23.494722 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.494702    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-69hq7"]
Jul 14 17:26:23.494831 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.494803    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:23.494932 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:23.494910    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:23.980126 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:23.980099    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwwj2_3b2853cf-fd5f-46a9-94a9-e73045273824/node-ca/0.log"
Jul 14 17:26:24.198236 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:24.198196    2567 generic.go:356] "Generic (PLEG): container finished" podID="04e6197d-898c-44ef-a067-708a5e78df09" containerID="0bb2d22edb99146bd666ced4eceb74ceb38635ee1bb0ee7946ec27f39abda1d0" exitCode=0
Jul 14 17:26:24.198803 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:24.198282    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerDied","Data":"0bb2d22edb99146bd666ced4eceb74ceb38635ee1bb0ee7946ec27f39abda1d0"}
Jul 14 17:26:25.045252 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:25.045220    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:25.045431 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:25.045316    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:25.045431 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:25.045418    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:25.045431 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:25.045322    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:25.045597 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:25.045318    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:25.045597 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:25.045539    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:26.206947 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:26.206909    2567 generic.go:356] "Generic (PLEG): container finished" podID="04e6197d-898c-44ef-a067-708a5e78df09" containerID="cbcfd279832ba1cca6d8a03844cb77da4be61143d45ef9ee4280e0da605d21db" exitCode=0
Jul 14 17:26:26.206947 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:26.206953    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerDied","Data":"cbcfd279832ba1cca6d8a03844cb77da4be61143d45ef9ee4280e0da605d21db"}
Jul 14 17:26:26.678411 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:26.678379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:26.678556 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:26.678416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:26.678556 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.678511    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 17:26:26.678648 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.678565    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret podName:f76fad34-a307-43b9-b220-75c29a93352b nodeName:}" failed. No retries permitted until 2026-07-14 17:26:58.678551539 +0000 UTC m=+65.222955315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret") pod "global-pull-secret-syncer-69hq7" (UID: "f76fad34-a307-43b9-b220-75c29a93352b") : object "kube-system"/"original-pull-secret" not registered
Jul 14 17:26:26.678648 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.678511    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:26:26.678648 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.678634    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs podName:2277aabd-4d56-4c55-9b26-90214ab70e6d nodeName:}" failed. No retries permitted until 2026-07-14 17:26:58.678623937 +0000 UTC m=+65.223027711 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs") pod "network-metrics-daemon-prqcj" (UID: "2277aabd-4d56-4c55-9b26-90214ab70e6d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 17:26:26.779140 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:26.779107    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:26.779310 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.779270    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 17:26:26.779310 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.779289    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 17:26:26.779310 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.779298    2567 projected.go:196] Error preparing data for projected volume kube-api-access-bwxbh for pod openshift-network-diagnostics/network-check-target-7knwp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:26:26.779456 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:26.779372    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh podName:6f2ddc55-0dc4-47b3-b219-34366a6434fc nodeName:}" failed. No retries permitted until 2026-07-14 17:26:58.779339225 +0000 UTC m=+65.323743019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bwxbh" (UniqueName: "kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh") pod "network-check-target-7knwp" (UID: "6f2ddc55-0dc4-47b3-b219-34366a6434fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 17:26:27.046036 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:27.045959    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:27.046036 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:27.045974    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:27.046253 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:27.045966    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:27.046253 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:27.046073    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:27.046253 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:27.046156    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:27.046253 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:27.046208    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:29.045634 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:29.045596    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:29.045634 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:29.045627    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:29.046122 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:29.045631    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:29.046122 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:29.045721    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:29.046122 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:29.045864    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:29.046122 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:29.045951    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:30.218330 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:30.218176    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerStarted","Data":"a017321ff6132c028a089ff534ad0493fdc4d1965ef4b7acc7783320079397b8"}
Jul 14 17:26:31.045962 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:31.045929    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:31.046128 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:31.046022    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:31.046128 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:31.046029    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:31.046128 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:31.046046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:31.046128 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:31.046107    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:31.046267 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:31.046171    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:31.222808 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:31.222775    2567 generic.go:356] "Generic (PLEG): container finished" podID="04e6197d-898c-44ef-a067-708a5e78df09" containerID="a017321ff6132c028a089ff534ad0493fdc4d1965ef4b7acc7783320079397b8" exitCode=0
Jul 14 17:26:31.223157 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:31.222827    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerDied","Data":"a017321ff6132c028a089ff534ad0493fdc4d1965ef4b7acc7783320079397b8"}
Jul 14 17:26:32.228438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:32.228406    2567 generic.go:356] "Generic (PLEG): container finished" podID="04e6197d-898c-44ef-a067-708a5e78df09" containerID="2c5fa0f5e8581b0b18c8db7c99320aa473a11353ba217d3cbf55b739f9b2ae5f" exitCode=0
Jul 14 17:26:32.228772 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:32.228468    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerDied","Data":"2c5fa0f5e8581b0b18c8db7c99320aa473a11353ba217d3cbf55b739f9b2ae5f"}
Jul 14 17:26:33.046027 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:33.045989    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:33.046181 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:33.046030    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:33.046181 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:33.046092    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:33.046181 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:33.046033    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:33.046181 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:33.046143    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:33.046337 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:33.046185    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:33.234622 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:33.234591    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zh27r" event={"ID":"04e6197d-898c-44ef-a067-708a5e78df09","Type":"ContainerStarted","Data":"5405d250b54aa1f8735c242596c22c51c928b10314d280964bb267f91d761793"}
Jul 14 17:26:33.256254 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:33.256159    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zh27r" podStartSLOduration=7.973276865 podStartE2EDuration="39.256139088s" totalImagesPullingTime="31.282862223s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:33.255774728 +0000 UTC m=+39.800178522" watchObservedRunningTime="2026-07-14 17:26:33.256139088 +0000 UTC m=+39.800542884"
Jul 14 17:26:35.045232 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:35.045192    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:35.045232 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:35.045221    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:35.045682 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:35.045246    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:35.045682 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:35.045314    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:35.045682 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:35.045367    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:35.045682 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:35.045451    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:37.045923 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:37.045889    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:37.045923 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:37.045915    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:37.046374 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:37.045992    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:37.046374 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:37.046040    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:37.046374 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:37.046112    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:37.046374 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:37.046150    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:39.045312 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:39.045279    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:39.045858 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:39.045283    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:39.045858 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:39.045410    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:39.045858 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:39.045283    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:39.045858 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:39.045461    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:39.045858 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:39.045508    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:41.045469 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:41.045430    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:41.045469 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:41.045468    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:41.045975 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:41.045468    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:41.045975 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:41.045537    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:41.045975 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:41.045652    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:41.045975 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:41.045703    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:43.046070 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:43.046043    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:43.046437 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:43.046046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:43.046437 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:43.046140    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-69hq7" podUID="f76fad34-a307-43b9-b220-75c29a93352b"
Jul 14 17:26:43.046437 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:43.046227    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-prqcj" podUID="2277aabd-4d56-4c55-9b26-90214ab70e6d"
Jul 14 17:26:43.046437 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:43.046046    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:43.046437 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:26:43.046302    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7knwp" podUID="6f2ddc55-0dc4-47b3-b219-34366a6434fc"
Jul 14 17:26:44.814206 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.814179    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-98.ec2.internal" event="NodeReady"
Jul 14 17:26:44.814558 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.814291    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 17:26:44.847090 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.847032    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64f56f78d5-gcxvq"]
Jul 14 17:26:44.859261 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.859241    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.861476 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.861452    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f56f78d5-gcxvq"]
Jul 14 17:26:44.862166 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.862141    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-smpv7\""
Jul 14 17:26:44.862166 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.862142    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 14 17:26:44.862361 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.862242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 14 17:26:44.862445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.862424    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 14 17:26:44.864608 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.864592    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p8vrm"]
Jul 14 17:26:44.869025 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.869007    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 14 17:26:44.877790 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.877769    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fgtjn"]
Jul 14 17:26:44.877903 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.877887    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:44.881196 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.881176    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 17:26:44.881446 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.881427    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 17:26:44.881812 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.881457    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2lrmp\""
Jul 14 17:26:44.881812 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.881530    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 17:26:44.889123 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.889106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p8vrm"]
Jul 14 17:26:44.889201 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.889131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fgtjn"]
Jul 14 17:26:44.889201 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.889152    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-79l4c"]
Jul 14 17:26:44.889267 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.889206    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:44.891625 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.891607    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 17:26:44.891700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.891660    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6ft2z\""
Jul 14 17:26:44.891737 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.891715    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 17:26:44.899093 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.899075    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:44.899844 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.899816    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79l4c"]
Jul 14 17:26:44.901446 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.901431    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nk7rg\""
Jul 14 17:26:44.901972 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.901851    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 17:26:44.901972 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.901898    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 17:26:44.902112 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.902038    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 17:26:44.902464 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.902446    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 17:26:44.921501 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921480    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b2hw\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-kube-api-access-6b2hw\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921594 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921506    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7a0cd7b-45ab-439f-9fba-962130dc2720-trusted-ca\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921594 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921527    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad73822d-c866-4d05-bb99-c05be31ca85f-config-volume\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:44.921594 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c36d5c-171f-4521-9568-6865dcf9c8e6-cert\") pod \"ingress-canary-p8vrm\" (UID: \"c5c36d5c-171f-4521-9568-6865dcf9c8e6\") " pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:44.921594 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921565    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad73822d-c866-4d05-bb99-c05be31ca85f-metrics-tls\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:44.921758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921625    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7a0cd7b-45ab-439f-9fba-962130dc2720-image-registry-private-configuration\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921651    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-registry-tls\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921674    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7a0cd7b-45ab-439f-9fba-962130dc2720-registry-certificates\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad73822d-c866-4d05-bb99-c05be31ca85f-tmp-dir\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:44.921758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921704    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn564\" (UniqueName: \"kubernetes.io/projected/ad73822d-c866-4d05-bb99-c05be31ca85f-kube-api-access-xn564\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:44.921758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921745    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7a0cd7b-45ab-439f-9fba-962130dc2720-ca-trust-extracted\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921924 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921772    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7a0cd7b-45ab-439f-9fba-962130dc2720-installation-pull-secrets\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921924 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921813    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-bound-sa-token\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:44.921924 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:44.921829    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnw87\" (UniqueName: \"kubernetes.io/projected/c5c36d5c-171f-4521-9568-6865dcf9c8e6-kube-api-access-fnw87\") pod \"ingress-canary-p8vrm\" (UID: \"c5c36d5c-171f-4521-9568-6865dcf9c8e6\") " pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:45.022810 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.022784    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7a0cd7b-45ab-439f-9fba-962130dc2720-installation-pull-secrets\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.022944 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.022817    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-crio-socket\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.022944 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.022935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-bound-sa-token\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.023027 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.022963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnw87\" (UniqueName: \"kubernetes.io/projected/c5c36d5c-171f-4521-9568-6865dcf9c8e6-kube-api-access-fnw87\") pod \"ingress-canary-p8vrm\" (UID: \"c5c36d5c-171f-4521-9568-6865dcf9c8e6\") " pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:45.023027 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.022982    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd9c\" (UniqueName: \"kubernetes.io/projected/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-kube-api-access-2dd9c\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.023027 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-data-volume\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.023140 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023033    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b2hw\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-kube-api-access-6b2hw\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.023140 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023052    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7a0cd7b-45ab-439f-9fba-962130dc2720-trusted-ca\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.023140 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.023140 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023095    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad73822d-c866-4d05-bb99-c05be31ca85f-config-volume\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.023302 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023169    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c36d5c-171f-4521-9568-6865dcf9c8e6-cert\") pod \"ingress-canary-p8vrm\" (UID: \"c5c36d5c-171f-4521-9568-6865dcf9c8e6\") " pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:45.023302 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023250    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad73822d-c866-4d05-bb99-c05be31ca85f-metrics-tls\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.023688 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7a0cd7b-45ab-439f-9fba-962130dc2720-image-registry-private-configuration\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.023861 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-registry-tls\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.023941 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7a0cd7b-45ab-439f-9fba-962130dc2720-registry-certificates\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.023941 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.023777    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad73822d-c866-4d05-bb99-c05be31ca85f-config-volume\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024105    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7a0cd7b-45ab-439f-9fba-962130dc2720-trusted-ca\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad73822d-c866-4d05-bb99-c05be31ca85f-tmp-dir\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024163    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn564\" (UniqueName: \"kubernetes.io/projected/ad73822d-c866-4d05-bb99-c05be31ca85f-kube-api-access-xn564\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7a0cd7b-45ab-439f-9fba-962130dc2720-ca-trust-extracted\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024220    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024437    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad73822d-c866-4d05-bb99-c05be31ca85f-tmp-dir\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024701    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7a0cd7b-45ab-439f-9fba-962130dc2720-registry-certificates\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.024762 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.024715    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7a0cd7b-45ab-439f-9fba-962130dc2720-ca-trust-extracted\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.027367 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.027320    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7a0cd7b-45ab-439f-9fba-962130dc2720-installation-pull-secrets\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.027367 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.027359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7a0cd7b-45ab-439f-9fba-962130dc2720-image-registry-private-configuration\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.027489 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.027334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad73822d-c866-4d05-bb99-c05be31ca85f-metrics-tls\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.027489 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.027423    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-registry-tls\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.027825 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.027807    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c36d5c-171f-4521-9568-6865dcf9c8e6-cert\") pod \"ingress-canary-p8vrm\" (UID: \"c5c36d5c-171f-4521-9568-6865dcf9c8e6\") " pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:45.031115 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.031090    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-bound-sa-token\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.031303 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.031286    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b2hw\" (UniqueName: \"kubernetes.io/projected/b7a0cd7b-45ab-439f-9fba-962130dc2720-kube-api-access-6b2hw\") pod \"image-registry-64f56f78d5-gcxvq\" (UID: \"b7a0cd7b-45ab-439f-9fba-962130dc2720\") " pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.031381 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.031330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnw87\" (UniqueName: \"kubernetes.io/projected/c5c36d5c-171f-4521-9568-6865dcf9c8e6-kube-api-access-fnw87\") pod \"ingress-canary-p8vrm\" (UID: \"c5c36d5c-171f-4521-9568-6865dcf9c8e6\") " pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:45.031657 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.031642    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn564\" (UniqueName: \"kubernetes.io/projected/ad73822d-c866-4d05-bb99-c05be31ca85f-kube-api-access-xn564\") pod \"dns-default-fgtjn\" (UID: \"ad73822d-c866-4d05-bb99-c05be31ca85f\") " pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.045721 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.045705    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:45.045808 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.045704    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:45.045894 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.045704    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:45.048276 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.048257    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 17:26:45.048376 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.048260    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 17:26:45.048529 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.048516    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ph225\""
Jul 14 17:26:45.048731 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.048715    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 17:26:45.048781 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.048715    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 17:26:45.048942 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.048929    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hwsmr\""
Jul 14 17:26:45.126510 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.126422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.126656 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.126588    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.126656 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.126626    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-crio-socket\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.126780 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.126673    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd9c\" (UniqueName: \"kubernetes.io/projected/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-kube-api-access-2dd9c\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.126780 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.126691    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-data-volume\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.126933 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.126906    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-crio-socket\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.127018 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.127001    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-data-volume\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.127080 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.127035    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.128748 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.128731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.134395 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.134362    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd9c\" (UniqueName: \"kubernetes.io/projected/c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897-kube-api-access-2dd9c\") pod \"insights-runtime-extractor-79l4c\" (UID: \"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897\") " pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.170381 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.170354    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:45.189449 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.189417    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p8vrm"
Jul 14 17:26:45.196432 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.196411    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:45.207547 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.207525    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-79l4c"
Jul 14 17:26:45.398801 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.398730    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p8vrm"]
Jul 14 17:26:45.401290 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.401267    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f56f78d5-gcxvq"]
Jul 14 17:26:45.402964 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:45.402943    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c36d5c_171f_4521_9568_6865dcf9c8e6.slice/crio-482ee38228cb2c0bde55d2d2e4b13c0d40a9c6404bf23010e824255f7f105f73 WatchSource:0}: Error finding container 482ee38228cb2c0bde55d2d2e4b13c0d40a9c6404bf23010e824255f7f105f73: Status 404 returned error can't find the container with id 482ee38228cb2c0bde55d2d2e4b13c0d40a9c6404bf23010e824255f7f105f73
Jul 14 17:26:45.404081 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:45.404059    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a0cd7b_45ab_439f_9fba_962130dc2720.slice/crio-36ea23c2bb1addef8b3afccb979fed8f1b87e1c6ddae846ed00e946ba60090b8 WatchSource:0}: Error finding container 36ea23c2bb1addef8b3afccb979fed8f1b87e1c6ddae846ed00e946ba60090b8: Status 404 returned error can't find the container with id 36ea23c2bb1addef8b3afccb979fed8f1b87e1c6ddae846ed00e946ba60090b8
Jul 14 17:26:45.622878 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.622849    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fgtjn"]
Jul 14 17:26:45.625981 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:45.625944    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad73822d_c866_4d05_bb99_c05be31ca85f.slice/crio-4d946eb0ad374f110665ee3fb419a522490d6bb8543d99d78ae9eb6ead4ebd76 WatchSource:0}: Error finding container 4d946eb0ad374f110665ee3fb419a522490d6bb8543d99d78ae9eb6ead4ebd76: Status 404 returned error can't find the container with id 4d946eb0ad374f110665ee3fb419a522490d6bb8543d99d78ae9eb6ead4ebd76
Jul 14 17:26:45.626087 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:45.626069    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-79l4c"]
Jul 14 17:26:45.628893 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:45.628858    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0abbd0d_6aaa_45c3_9e4e_b44c14ca7897.slice/crio-572d742f7e5eccd5f126add4a6ced1280950d23c26de58ea23bcb20bb7cbbe6b WatchSource:0}: Error finding container 572d742f7e5eccd5f126add4a6ced1280950d23c26de58ea23bcb20bb7cbbe6b: Status 404 returned error can't find the container with id 572d742f7e5eccd5f126add4a6ced1280950d23c26de58ea23bcb20bb7cbbe6b
Jul 14 17:26:46.263482 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.263449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fgtjn" event={"ID":"ad73822d-c866-4d05-bb99-c05be31ca85f","Type":"ContainerStarted","Data":"4d946eb0ad374f110665ee3fb419a522490d6bb8543d99d78ae9eb6ead4ebd76"}
Jul 14 17:26:46.264641 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.264613    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p8vrm" event={"ID":"c5c36d5c-171f-4521-9568-6865dcf9c8e6","Type":"ContainerStarted","Data":"482ee38228cb2c0bde55d2d2e4b13c0d40a9c6404bf23010e824255f7f105f73"}
Jul 14 17:26:46.270207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.270174    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq" event={"ID":"b7a0cd7b-45ab-439f-9fba-962130dc2720","Type":"ContainerStarted","Data":"1f952a417d8c584b9b5d0258e638536e7324383334f6cc7a66f9e5d62775fcda"}
Jul 14 17:26:46.270329 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.270259    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq" event={"ID":"b7a0cd7b-45ab-439f-9fba-962130dc2720","Type":"ContainerStarted","Data":"36ea23c2bb1addef8b3afccb979fed8f1b87e1c6ddae846ed00e946ba60090b8"}
Jul 14 17:26:46.270329 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.270289    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:26:46.271661 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.271630    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79l4c" event={"ID":"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897","Type":"ContainerStarted","Data":"572d742f7e5eccd5f126add4a6ced1280950d23c26de58ea23bcb20bb7cbbe6b"}
Jul 14 17:26:46.290007 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:46.289956    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq" podStartSLOduration=2.289938511 podStartE2EDuration="2.289938511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:46.288117525 +0000 UTC m=+52.832521319" watchObservedRunningTime="2026-07-14 17:26:46.289938511 +0000 UTC m=+52.834342287"
Jul 14 17:26:51.288940 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.288902    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fgtjn" event={"ID":"ad73822d-c866-4d05-bb99-c05be31ca85f","Type":"ContainerStarted","Data":"f7ccc14f131f2fc621b970f96c6b2c60275f6edc3ea957c7c0e4ac8d9aecf0da"}
Jul 14 17:26:51.289427 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.289012    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fgtjn" event={"ID":"ad73822d-c866-4d05-bb99-c05be31ca85f","Type":"ContainerStarted","Data":"fa397da853fc61946c3e2f19ebb493ee2f3353642d1cd4d0fe3e7993e400cd47"}
Jul 14 17:26:51.289427 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.289045    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:26:51.290516 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.290486    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p8vrm" event={"ID":"c5c36d5c-171f-4521-9568-6865dcf9c8e6","Type":"ContainerStarted","Data":"aef13511494c2d22cf44ea96166af981913a00e7a00b331fdfd92a5c95aae7e0"}
Jul 14 17:26:51.291814 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.291792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79l4c" event={"ID":"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897","Type":"ContainerStarted","Data":"09bccf427646af480042a269e5073285a8945e9cba6c2bc6326c11dadf9b9ce9"}
Jul 14 17:26:51.305876 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.305804    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fgtjn" podStartSLOduration=1.8874558750000001 podStartE2EDuration="7.3057923s" totalImagesPullingTime="5.418336425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:51.304425627 +0000 UTC m=+57.848829419" watchObservedRunningTime="2026-07-14 17:26:51.3057923 +0000 UTC m=+57.850196094"
Jul 14 17:26:51.318471 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:51.318422    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p8vrm" podStartSLOduration=1.676113009 podStartE2EDuration="7.318410835s" totalImagesPullingTime="5.642297826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:51.318400348 +0000 UTC m=+57.862804141" watchObservedRunningTime="2026-07-14 17:26:51.318410835 +0000 UTC m=+57.862814663"
Jul 14 17:26:52.297675 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:52.297633    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79l4c" event={"ID":"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897","Type":"ContainerStarted","Data":"90261b1fa994c12df59be577cce8ef97a3d5cc9a8208e630e399677fbcd08c72"}
Jul 14 17:26:55.220394 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:55.220366    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pqtvp"
Jul 14 17:26:55.312032 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:55.311999    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-79l4c" event={"ID":"c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897","Type":"ContainerStarted","Data":"173ec6e6576b48d1aabeedcc1445b538023ab904f23cf01363d6e42e67dc790e"}
Jul 14 17:26:55.329845 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:55.329790    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-79l4c" podStartSLOduration=2.806415395 podStartE2EDuration="11.329776084s" totalImagesPullingTime="8.523360689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:26:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:26:55.328488645 +0000 UTC m=+61.872892449" watchObservedRunningTime="2026-07-14 17:26:55.329776084 +0000 UTC m=+61.874179877"
Jul 14 17:26:58.737505 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.737469    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:58.737505 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.737513    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:58.740008 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.739988    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 17:26:58.740008 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.740000    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 17:26:58.750422 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.750392    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2277aabd-4d56-4c55-9b26-90214ab70e6d-metrics-certs\") pod \"network-metrics-daemon-prqcj\" (UID: \"2277aabd-4d56-4c55-9b26-90214ab70e6d\") " pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:58.750530 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.750476    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f76fad34-a307-43b9-b220-75c29a93352b-original-pull-secret\") pod \"global-pull-secret-syncer-69hq7\" (UID: \"f76fad34-a307-43b9-b220-75c29a93352b\") " pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:58.838230 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.838195    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:58.840919 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.840900    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 17:26:58.851172 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.851149    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 17:26:58.856595 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.856579    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hwsmr\""
Jul 14 17:26:58.861188 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.861173    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxbh\" (UniqueName: \"kubernetes.io/projected/6f2ddc55-0dc4-47b3-b219-34366a6434fc-kube-api-access-bwxbh\") pod \"network-check-target-7knwp\" (UID: \"6f2ddc55-0dc4-47b3-b219-34366a6434fc\") " pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:58.864965 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.864946    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-69hq7"
Jul 14 17:26:58.865072 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.864944    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-prqcj"
Jul 14 17:26:58.990624 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:58.990560    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-prqcj"]
Jul 14 17:26:58.993716 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:58.993690    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2277aabd_4d56_4c55_9b26_90214ab70e6d.slice/crio-961a2d1cc8960c07aa6cf8fe15a8d47839bd1258aa2236e9ecd185ff2a2b3849 WatchSource:0}: Error finding container 961a2d1cc8960c07aa6cf8fe15a8d47839bd1258aa2236e9ecd185ff2a2b3849: Status 404 returned error can't find the container with id 961a2d1cc8960c07aa6cf8fe15a8d47839bd1258aa2236e9ecd185ff2a2b3849
Jul 14 17:26:59.007468 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.007448    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-69hq7"]
Jul 14 17:26:59.009998 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:59.009978    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76fad34_a307_43b9_b220_75c29a93352b.slice/crio-d0cb0fc5a58c426983451917f936558f9677aab2a57a4ad5395f2809b667cb73 WatchSource:0}: Error finding container d0cb0fc5a58c426983451917f936558f9677aab2a57a4ad5395f2809b667cb73: Status 404 returned error can't find the container with id d0cb0fc5a58c426983451917f936558f9677aab2a57a4ad5395f2809b667cb73
Jul 14 17:26:59.163071 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.163041    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ph225\""
Jul 14 17:26:59.171166 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.171144    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:26:59.282703 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.282674    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7knwp"]
Jul 14 17:26:59.285204 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:26:59.285179    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2ddc55_0dc4_47b3_b219_34366a6434fc.slice/crio-716879c5d0d06434334d35df781fa0041fc4c7d4c5f53b4cc1e5d4e5ab213e47 WatchSource:0}: Error finding container 716879c5d0d06434334d35df781fa0041fc4c7d4c5f53b4cc1e5d4e5ab213e47: Status 404 returned error can't find the container with id 716879c5d0d06434334d35df781fa0041fc4c7d4c5f53b4cc1e5d4e5ab213e47
Jul 14 17:26:59.327433 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.327400    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7knwp" event={"ID":"6f2ddc55-0dc4-47b3-b219-34366a6434fc","Type":"ContainerStarted","Data":"716879c5d0d06434334d35df781fa0041fc4c7d4c5f53b4cc1e5d4e5ab213e47"}
Jul 14 17:26:59.328478 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.328450    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prqcj" event={"ID":"2277aabd-4d56-4c55-9b26-90214ab70e6d","Type":"ContainerStarted","Data":"961a2d1cc8960c07aa6cf8fe15a8d47839bd1258aa2236e9ecd185ff2a2b3849"}
Jul 14 17:26:59.329398 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:26:59.329379    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-69hq7" event={"ID":"f76fad34-a307-43b9-b220-75c29a93352b","Type":"ContainerStarted","Data":"d0cb0fc5a58c426983451917f936558f9677aab2a57a4ad5395f2809b667cb73"}
Jul 14 17:27:00.338820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:00.338779    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prqcj" event={"ID":"2277aabd-4d56-4c55-9b26-90214ab70e6d","Type":"ContainerStarted","Data":"cf4f1011e5a598431f3ec2efda6a6f1eaa61b168dfe038207ce92093ff287723"}
Jul 14 17:27:00.339278 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:00.338861    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-prqcj" event={"ID":"2277aabd-4d56-4c55-9b26-90214ab70e6d","Type":"ContainerStarted","Data":"0a7685be6143e10c870c5f0e6f5ae46cd151dacd77526493a454c37de84dc298"}
Jul 14 17:27:00.355205 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:00.355157    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-prqcj" podStartSLOduration=65.38726078 podStartE2EDuration="1m6.355141656s" totalImagesPullingTime="967.880876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:00.353624966 +0000 UTC m=+66.898028762" watchObservedRunningTime="2026-07-14 17:27:00.355141656 +0000 UTC m=+66.899545449"
Jul 14 17:27:01.300284 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:01.300252    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fgtjn"
Jul 14 17:27:02.346931 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:02.346894    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7knwp" event={"ID":"6f2ddc55-0dc4-47b3-b219-34366a6434fc","Type":"ContainerStarted","Data":"c31122fc0aeb3f523625b32cfcb2d3108da000ef1c34417b553a87c36df4ced8"}
Jul 14 17:27:02.347325 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:02.346980    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:27:02.361388 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:02.361330    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7knwp" podStartSLOduration=65.476042202 podStartE2EDuration="1m8.36131067s" totalImagesPullingTime="2.885268468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:02.361266409 +0000 UTC m=+68.905670224" watchObservedRunningTime="2026-07-14 17:27:02.36131067 +0000 UTC m=+68.905714489"
Jul 14 17:27:04.355676 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:04.355626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-69hq7" event={"ID":"f76fad34-a307-43b9-b220-75c29a93352b","Type":"ContainerStarted","Data":"5f3fc8eb7bd50385f7aaf75ebea553688cb97d4b85ab97545f08d962cbf393d1"}
Jul 14 17:27:04.372263 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:04.372208    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-69hq7" podStartSLOduration=65.228643399 podStartE2EDuration="1m10.372192953s" totalImagesPullingTime="5.143549554s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 17:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:04.372040579 +0000 UTC m=+70.916444363" watchObservedRunningTime="2026-07-14 17:27:04.372192953 +0000 UTC m=+70.916596747"
Jul 14 17:27:05.174651 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.174615    2567 patch_prober.go:28] interesting pod/image-registry-64f56f78d5-gcxvq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 14 17:27:05.174790 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.174666    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq" podUID="b7a0cd7b-45ab-439f-9fba-962130dc2720" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 17:27:05.529282 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.529210    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"]
Jul 14 17:27:05.530835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.530819    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.533686 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.533666    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 14 17:27:05.535041 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.535019    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 17:27:05.535041 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.535038    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-c6vlw\""
Jul 14 17:27:05.535210 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.535080    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 17:27:05.535210 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.535083    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 17:27:05.535210 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.535121    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 14 17:27:05.542368 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.542328    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"]
Jul 14 17:27:05.553440 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.553420    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"]
Jul 14 17:27:05.555267 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.555254    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.557666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.557636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 14 17:27:05.557666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.557653    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 14 17:27:05.557791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.557708    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-khz96\""
Jul 14 17:27:05.557791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.557717    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 14 17:27:05.571192 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.571168    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qpbk9"]
Jul 14 17:27:05.572873 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.572856    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"]
Jul 14 17:27:05.572962 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.572939    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.575584 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.575556    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 17:27:05.575669 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.575589    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zfrgx\""
Jul 14 17:27:05.575669 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.575642    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 17:27:05.575669 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.575651    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 17:27:05.685387 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685425    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685442    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnd9\" (UniqueName: \"kubernetes.io/projected/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-kube-api-access-vcnd9\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685463    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-sys\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685525    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gkbk\" (UniqueName: \"kubernetes.io/projected/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-api-access-5gkbk\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685542    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-textfile\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.685620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685603    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685663    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685696    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpbm\" (UniqueName: \"kubernetes.io/projected/208c5abd-9e61-45ba-8575-69fdfa665e5b-kube-api-access-dtpbm\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685716    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685765    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d76197-e02d-4953-ac8c-08c45813a7fe-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685782    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-tls\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685825    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29d76197-e02d-4953-ac8c-08c45813a7fe-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-wtmp\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.685889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685888    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-root\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.686115 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.685905    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/208c5abd-9e61-45ba-8575-69fdfa665e5b-metrics-client-ca\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.786632 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786544    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.786632 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpbm\" (UniqueName: \"kubernetes.io/projected/208c5abd-9e61-45ba-8575-69fdfa665e5b-kube-api-access-dtpbm\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.786632 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.786831 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:05.786695    2567 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 14 17:27:05.786831 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d76197-e02d-4953-ac8c-08c45813a7fe-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.786831 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:05.786761    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-tls podName:6a6d3c32-51aa-42c7-acfb-d1c804cf783a nodeName:}" failed. No retries permitted until 2026-07-14 17:27:06.286744725 +0000 UTC m=+72.831148497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-lb8kv" (UID: "6a6d3c32-51aa-42c7-acfb-d1c804cf783a") : secret "openshift-state-metrics-tls" not found
Jul 14 17:27:05.786831 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786782    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-tls\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.786831 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29d76197-e02d-4953-ac8c-08c45813a7fe-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-wtmp\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786856    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-root\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786880    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/208c5abd-9e61-45ba-8575-69fdfa665e5b-metrics-client-ca\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786912    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786956    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-root\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787058 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786995    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-wtmp\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.786962    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787214    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnd9\" (UniqueName: \"kubernetes.io/projected/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-kube-api-access-vcnd9\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-sys\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787252    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-accelerators-collector-config\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gkbk\" (UniqueName: \"kubernetes.io/projected/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-api-access-5gkbk\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787299    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/29d76197-e02d-4953-ac8c-08c45813a7fe-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787321    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-textfile\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787381    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/208c5abd-9e61-45ba-8575-69fdfa665e5b-sys\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787386    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.787425 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787417    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.787889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787445    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/208c5abd-9e61-45ba-8575-69fdfa665e5b-metrics-client-ca\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.787889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.787778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-textfile\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.788185 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.788156    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.788381 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.788234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.788649 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.788626    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29d76197-e02d-4953-ac8c-08c45813a7fe-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.789478 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.789458    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-tls\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.789681 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.789657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.789948 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.789923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/208c5abd-9e61-45ba-8575-69fdfa665e5b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.790053 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.790037    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.790144 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.790127    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.794478 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.794453    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpbm\" (UniqueName: \"kubernetes.io/projected/208c5abd-9e61-45ba-8575-69fdfa665e5b-kube-api-access-dtpbm\") pod \"node-exporter-qpbk9\" (UID: \"208c5abd-9e61-45ba-8575-69fdfa665e5b\") " pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.794935 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.794919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnd9\" (UniqueName: \"kubernetes.io/projected/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-kube-api-access-vcnd9\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:05.795137 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.795118    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gkbk\" (UniqueName: \"kubernetes.io/projected/29d76197-e02d-4953-ac8c-08c45813a7fe-kube-api-access-5gkbk\") pod \"kube-state-metrics-785778ff74-tqrqg\" (UID: \"29d76197-e02d-4953-ac8c-08c45813a7fe\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.863714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.863679    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"
Jul 14 17:27:05.881240 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.881219    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qpbk9"
Jul 14 17:27:05.892579 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:05.892554    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208c5abd_9e61_45ba_8575_69fdfa665e5b.slice/crio-c745f179e9e54907577a7a06f26a08b0d326d182987a437718cf6bc4207efdf7 WatchSource:0}: Error finding container c745f179e9e54907577a7a06f26a08b0d326d182987a437718cf6bc4207efdf7: Status 404 returned error can't find the container with id c745f179e9e54907577a7a06f26a08b0d326d182987a437718cf6bc4207efdf7
Jul 14 17:27:05.985131 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:05.985106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-tqrqg"]
Jul 14 17:27:05.987410 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:05.987386    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d76197_e02d_4953_ac8c_08c45813a7fe.slice/crio-ed2b4e15dbe2dba15c32f0373caf195651968b9f1c87cb6faac9cb3470e08b23 WatchSource:0}: Error finding container ed2b4e15dbe2dba15c32f0373caf195651968b9f1c87cb6faac9cb3470e08b23: Status 404 returned error can't find the container with id ed2b4e15dbe2dba15c32f0373caf195651968b9f1c87cb6faac9cb3470e08b23
Jul 14 17:27:06.291504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.291470    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:06.293683 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.293665    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a6d3c32-51aa-42c7-acfb-d1c804cf783a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-lb8kv\" (UID: \"6a6d3c32-51aa-42c7-acfb-d1c804cf783a\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:06.366072 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.366036    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg" event={"ID":"29d76197-e02d-4953-ac8c-08c45813a7fe","Type":"ContainerStarted","Data":"ed2b4e15dbe2dba15c32f0373caf195651968b9f1c87cb6faac9cb3470e08b23"}
Jul 14 17:27:06.367044 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.367022    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qpbk9" event={"ID":"208c5abd-9e61-45ba-8575-69fdfa665e5b","Type":"ContainerStarted","Data":"c745f179e9e54907577a7a06f26a08b0d326d182987a437718cf6bc4207efdf7"}
Jul 14 17:27:06.439510 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.439481    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"
Jul 14 17:27:06.566082 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.566004    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv"]
Jul 14 17:27:06.569395 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:06.569364    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6d3c32_51aa_42c7_acfb_d1c804cf783a.slice/crio-8c337d8058a42af27d9309e63a0a42fa6203374972e983b7a36162ac8f24b960 WatchSource:0}: Error finding container 8c337d8058a42af27d9309e63a0a42fa6203374972e983b7a36162ac8f24b960: Status 404 returned error can't find the container with id 8c337d8058a42af27d9309e63a0a42fa6203374972e983b7a36162ac8f24b960
Jul 14 17:27:06.576559 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.576534    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 17:27:06.580512 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.580492    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.583037 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.582803    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 14 17:27:06.583037 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.582851    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-klxtj\""
Jul 14 17:27:06.583714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.583362    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 14 17:27:06.583714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.583424    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 14 17:27:06.583714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.583437    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 14 17:27:06.583714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.583449    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 14 17:27:06.583714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.583587    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 14 17:27:06.584719 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.584221    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 14 17:27:06.584719 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.584227    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 14 17:27:06.584719 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.584459    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 14 17:27:06.592931 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.592908    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 17:27:06.693104 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693079    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693113    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693136    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693171    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693196    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fg4w\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-kube-api-access-2fg4w\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-web-config\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693287 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693576 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693293    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693576 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-config-out\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693576 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693419    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.693576 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.693445    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794047 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794217 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794063    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-web-config\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794217 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794088    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794217 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794110    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794390 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:06.794307    2567 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 14 17:27:06.794449 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:06.794391    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls podName:f006b60b-4919-4801-8163-1ff15765dc1e nodeName:}" failed. No retries permitted until 2026-07-14 17:27:07.294368869 +0000 UTC m=+73.838772646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e") : secret "alertmanager-main-tls" not found
Jul 14 17:27:06.794524 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794493    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-config-out\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794582 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794648 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794584    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794648 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794748 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794668    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794748 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794695    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794748 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794887 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.794887 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.794789    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fg4w\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-kube-api-access-2fg4w\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.795696 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.795296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.795696 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:06.795441    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle podName:f006b60b-4919-4801-8163-1ff15765dc1e nodeName:}" failed. No retries permitted until 2026-07-14 17:27:07.295420552 +0000 UTC m=+73.839824325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e") : configmap references non-existent config key: ca-bundle.crt
Jul 14 17:27:06.795696 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.795565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.797220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.797192    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-web-config\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.797303 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.797244    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.797871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.797849    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-config-out\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.798002 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.797955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.798616 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.798407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.799030 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.798847    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.799559 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.799533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.799811 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.799795    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:06.802661 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:06.802640    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fg4w\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-kube-api-access-2fg4w\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:07.280722 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.280695    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64f56f78d5-gcxvq"
Jul 14 17:27:07.298534 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.298481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:07.298687 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.298592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:07.299556 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.299528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:07.301223 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.301196    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:07.373797 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.373758    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv" event={"ID":"6a6d3c32-51aa-42c7-acfb-d1c804cf783a","Type":"ContainerStarted","Data":"5245edd0645a2a09c3bb1b9fdedb995c4cf040b54d1ea11ba6dd044d862f4493"}
Jul 14 17:27:07.373944 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.373827    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv" event={"ID":"6a6d3c32-51aa-42c7-acfb-d1c804cf783a","Type":"ContainerStarted","Data":"a88ae34255fda032681a2e5da8f81189d25a67a76dc80115b832773573367774"}
Jul 14 17:27:07.373944 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.373855    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv" event={"ID":"6a6d3c32-51aa-42c7-acfb-d1c804cf783a","Type":"ContainerStarted","Data":"8c337d8058a42af27d9309e63a0a42fa6203374972e983b7a36162ac8f24b960"}
Jul 14 17:27:07.500328 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.500294    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:27:07.823815 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:07.823783    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 17:27:07.828111 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:07.828080    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf006b60b_4919_4801_8163_1ff15765dc1e.slice/crio-273df5a8797897b7c4a9fafe6fa0f4f3ddb3cc6367abd474ed641d6b84a8ceb7 WatchSource:0}: Error finding container 273df5a8797897b7c4a9fafe6fa0f4f3ddb3cc6367abd474ed641d6b84a8ceb7: Status 404 returned error can't find the container with id 273df5a8797897b7c4a9fafe6fa0f4f3ddb3cc6367abd474ed641d6b84a8ceb7
Jul 14 17:27:08.378881 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.378851    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv" event={"ID":"6a6d3c32-51aa-42c7-acfb-d1c804cf783a","Type":"ContainerStarted","Data":"f20ba1b56dbb1bf599dae36a8f5fb1ca10fdd5835ac0198a82d75d8d7a836b52"}
Jul 14 17:27:08.381093 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.381066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg" event={"ID":"29d76197-e02d-4953-ac8c-08c45813a7fe","Type":"ContainerStarted","Data":"fcfe6809817c84212fd4792037a7967d4bec5a2179766e67dfa09f4c83935243"}
Jul 14 17:27:08.381213 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.381131    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg" event={"ID":"29d76197-e02d-4953-ac8c-08c45813a7fe","Type":"ContainerStarted","Data":"06695adfd0469d4888c7fad21fa380d9c46f60b016f718de92690b962b596324"}
Jul 14 17:27:08.381213 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.381164    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg" event={"ID":"29d76197-e02d-4953-ac8c-08c45813a7fe","Type":"ContainerStarted","Data":"b508670b6803983ca39ba2035bcd9f3c45c1e45b01f3c729e28f42836e773e49"}
Jul 14 17:27:08.382529 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.382504    2567 generic.go:356] "Generic (PLEG): container finished" podID="208c5abd-9e61-45ba-8575-69fdfa665e5b" containerID="b8826bfe658c7ef0a6d8243c465ba394a460960ceb76c2ef1230cdbdcc010168" exitCode=0
Jul 14 17:27:08.382624 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.382595    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qpbk9" event={"ID":"208c5abd-9e61-45ba-8575-69fdfa665e5b","Type":"ContainerDied","Data":"b8826bfe658c7ef0a6d8243c465ba394a460960ceb76c2ef1230cdbdcc010168"}
Jul 14 17:27:08.383691 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.383672    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"273df5a8797897b7c4a9fafe6fa0f4f3ddb3cc6367abd474ed641d6b84a8ceb7"}
Jul 14 17:27:08.395426 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.395371    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-lb8kv" podStartSLOduration=1.792538871 podStartE2EDuration="3.395335889s" totalImagesPullingTime="1.602797018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:08.395041913 +0000 UTC m=+74.939445709" watchObservedRunningTime="2026-07-14 17:27:08.395335889 +0000 UTC m=+74.939739683"
Jul 14 17:27:08.418947 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:08.418907    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-tqrqg" podStartSLOduration=1.7348369670000001 podStartE2EDuration="3.418891383s" totalImagesPullingTime="1.684054416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:08.418060746 +0000 UTC m=+74.962464541" watchObservedRunningTime="2026-07-14 17:27:08.418891383 +0000 UTC m=+74.963295176"
Jul 14 17:27:09.389980 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.389893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qpbk9" event={"ID":"208c5abd-9e61-45ba-8575-69fdfa665e5b","Type":"ContainerStarted","Data":"c5c08e83f019be0702e5515031804cddba2d66e9959160c7bef57590d430d65e"}
Jul 14 17:27:09.389980 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.389946    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qpbk9" event={"ID":"208c5abd-9e61-45ba-8575-69fdfa665e5b","Type":"ContainerStarted","Data":"d9ec3bce7111f870f72de59016ef2127a9555ac458fffb85a804545bba377eb3"}
Jul 14 17:27:09.391391 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.391370    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60" exitCode=0
Jul 14 17:27:09.391456 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.391433    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"}
Jul 14 17:27:09.409757 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.409711    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qpbk9" podStartSLOduration=2.524382552 podStartE2EDuration="4.409697518s" totalImagesPullingTime="1.780866492s" totalInitContainerRuntime="104.448474ms" isStatefulPod=true podCreationTimestamp="2026-07-14 17:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:09.40809806 +0000 UTC m=+75.952501854" watchObservedRunningTime="2026-07-14 17:27:09.409697518 +0000 UTC m=+75.954101311"
Jul 14 17:27:09.948906 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.948870    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68cc46949c-2kn2s"]
Jul 14 17:27:09.952007 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.951991    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:09.954661 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.954640    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 14 17:27:09.955720 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.955700    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ei9bi9h6fusul\""
Jul 14 17:27:09.955818 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.955760    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 17:27:09.955818 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.955699    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 14 17:27:09.955818 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.955708    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gngj4\""
Jul 14 17:27:09.955818 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.955803    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 14 17:27:09.962055 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:09.962033    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cc46949c-2kn2s"]
Jul 14 17:27:10.125708 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125669    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-secret-metrics-server-tls\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.125708 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125709    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-client-ca-bundle\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.125904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125757    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-secret-metrics-server-client-certs\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.125904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125781    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2dee9203-7020-4600-a8e6-898ca56d7411-metrics-server-audit-profiles\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.125904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125809    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dee9203-7020-4600-a8e6-898ca56d7411-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.125904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125828    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vsj\" (UniqueName: \"kubernetes.io/projected/2dee9203-7020-4600-a8e6-898ca56d7411-kube-api-access-24vsj\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.126025 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.125901    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2dee9203-7020-4600-a8e6-898ca56d7411-audit-log\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.226973 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.226888    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2dee9203-7020-4600-a8e6-898ca56d7411-audit-log\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.226973 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.226964    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-secret-metrics-server-tls\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.226997    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-client-ca-bundle\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.227045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-secret-metrics-server-client-certs\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.227071    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2dee9203-7020-4600-a8e6-898ca56d7411-metrics-server-audit-profiles\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.227112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dee9203-7020-4600-a8e6-898ca56d7411-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.227137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vsj\" (UniqueName: \"kubernetes.io/projected/2dee9203-7020-4600-a8e6-898ca56d7411-kube-api-access-24vsj\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227431 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.227306    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2dee9203-7020-4600-a8e6-898ca56d7411-audit-log\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.227889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.227862    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dee9203-7020-4600-a8e6-898ca56d7411-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.228129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.228106    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2dee9203-7020-4600-a8e6-898ca56d7411-metrics-server-audit-profiles\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.229507 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.229486    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-secret-metrics-server-tls\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.229589 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.229553    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-secret-metrics-server-client-certs\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.229589 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.229575    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2dee9203-7020-4600-a8e6-898ca56d7411-client-ca-bundle\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.235527 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.235509    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vsj\" (UniqueName: \"kubernetes.io/projected/2dee9203-7020-4600-a8e6-898ca56d7411-kube-api-access-24vsj\") pod \"metrics-server-68cc46949c-2kn2s\" (UID: \"2dee9203-7020-4600-a8e6-898ca56d7411\") " pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.260410 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.260389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:10.321067 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.320398    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"]
Jul 14 17:27:10.325036 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.325018    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:10.328298 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.328269    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 14 17:27:10.328695 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.328678    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-n28xb\""
Jul 14 17:27:10.341107 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.341079    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"]
Jul 14 17:27:10.389507 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.389477    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68cc46949c-2kn2s"]
Jul 14 17:27:10.392448 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:10.392410    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dee9203_7020_4600_a8e6_898ca56d7411.slice/crio-36a17fd37b138b7aae419594d54b6d5973fb19f5ef4ed16124a8790dca0f575c WatchSource:0}: Error finding container 36a17fd37b138b7aae419594d54b6d5973fb19f5ef4ed16124a8790dca0f575c: Status 404 returned error can't find the container with id 36a17fd37b138b7aae419594d54b6d5973fb19f5ef4ed16124a8790dca0f575c
Jul 14 17:27:10.428992 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.428958    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtxn\" (UniqueName: \"kubernetes.io/projected/055a5c57-6812-44e4-995c-3d645338393f-kube-api-access-nvtxn\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:10.429160 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.429058    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/055a5c57-6812-44e4-995c-3d645338393f-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:10.529753 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.529665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtxn\" (UniqueName: \"kubernetes.io/projected/055a5c57-6812-44e4-995c-3d645338393f-kube-api-access-nvtxn\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:10.529753 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.529729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/055a5c57-6812-44e4-995c-3d645338393f-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:10.529974 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:10.529813    2567 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 14 17:27:10.529974 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:27:10.529861    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/055a5c57-6812-44e4-995c-3d645338393f-monitoring-plugin-cert podName:055a5c57-6812-44e4-995c-3d645338393f nodeName:}" failed. No retries permitted until 2026-07-14 17:27:11.029847118 +0000 UTC m=+77.574250890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/055a5c57-6812-44e4-995c-3d645338393f-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-r7575" (UID: "055a5c57-6812-44e4-995c-3d645338393f") : secret "monitoring-plugin-cert" not found
Jul 14 17:27:10.537822 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:10.537804    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtxn\" (UniqueName: \"kubernetes.io/projected/055a5c57-6812-44e4-995c-3d645338393f-kube-api-access-nvtxn\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:11.034640 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.034598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/055a5c57-6812-44e4-995c-3d645338393f-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:11.037399 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.037369    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/055a5c57-6812-44e4-995c-3d645338393f-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-r7575\" (UID: \"055a5c57-6812-44e4-995c-3d645338393f\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:11.234396 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.234366    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:11.360099 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.359875    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"]
Jul 14 17:27:11.362488 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:11.362456    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055a5c57_6812_44e4_995c_3d645338393f.slice/crio-78a077d8f8e0a517fcf9c7b443cf7e7afe803157b39373c3043119fe4bfa0b51 WatchSource:0}: Error finding container 78a077d8f8e0a517fcf9c7b443cf7e7afe803157b39373c3043119fe4bfa0b51: Status 404 returned error can't find the container with id 78a077d8f8e0a517fcf9c7b443cf7e7afe803157b39373c3043119fe4bfa0b51
Jul 14 17:27:11.401315 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.401283    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s" event={"ID":"2dee9203-7020-4600-a8e6-898ca56d7411","Type":"ContainerStarted","Data":"36a17fd37b138b7aae419594d54b6d5973fb19f5ef4ed16124a8790dca0f575c"}
Jul 14 17:27:11.402662 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.402643    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575" event={"ID":"055a5c57-6812-44e4-995c-3d645338393f","Type":"ContainerStarted","Data":"78a077d8f8e0a517fcf9c7b443cf7e7afe803157b39373c3043119fe4bfa0b51"}
Jul 14 17:27:11.742585 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.742553    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:27:11.745945 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.745923    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.748514 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.748490    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 17:27:11.748646 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.748515    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 17:27:11.748646 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.748492    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 17:27:11.748646 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.748544    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3bkh8es2oqpa0\""
Jul 14 17:27:11.748646 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.748555    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 17:27:11.748965 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.748952    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 17:27:11.749061 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749039    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 17:27:11.749124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749072    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 17:27:11.749201 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749181    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 17:27:11.749328 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749313    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 17:27:11.749394 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749339    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 17:27:11.749820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749799    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 17:27:11.749898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749819    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 17:27:11.749898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749838    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 17:27:11.749898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.749803    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-btvtb\""
Jul 14 17:27:11.751776 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.751759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 17:27:11.762537 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.762513    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:27:11.842279 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842253    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842399 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842287    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842399 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842311    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842491 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59j5w\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-kube-api-access-59j5w\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842491 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842446    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842491 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842487    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842586 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842528    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842628 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842580    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-config\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842628 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842607    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842694 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842637    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842694 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842665    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842771 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842690    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842771 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842720    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842771 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842745    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842792    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842842    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.842891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842874    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.843042 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842897    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.843042 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.842921    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.943757 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943731    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.943881 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.943881 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-config\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.943881 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943835    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.943881 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943855    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.943881 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943878    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943901    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943958    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.943988    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944040    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944068    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944129 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944103    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944138    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944175    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944198    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944252    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59j5w\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-kube-api-access-59j5w\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.944846 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.944821    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.946553 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.945668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.946553 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.946175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.946893 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.946778    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.947235 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.947205    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.947393 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.947373    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.949405 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.949040    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-config-out\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.949405 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.949375    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.950267 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.950231    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.950430 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.950399    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.950879 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.950761    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.951143 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.951120    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.951255 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.951190    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-web-config\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.951921 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.951897    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.952041 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.952017    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.952275 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.952256    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-config\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.952315 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.952274    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.953878 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.953854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.954271 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.954250    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:11.955797 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:11.955782    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59j5w\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-kube-api-access-59j5w\") pod \"prometheus-k8s-0\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:12.056704 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.056659    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:12.226175 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.226060    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:27:12.229439 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:27:12.229407    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e4e93c_3103_4538_b713_dff5ec8228f8.slice/crio-3d273eb94b58401e3d63d38bf5b329e0fb17c9d08ca6ef61e370f62fab405aac WatchSource:0}: Error finding container 3d273eb94b58401e3d63d38bf5b329e0fb17c9d08ca6ef61e370f62fab405aac: Status 404 returned error can't find the container with id 3d273eb94b58401e3d63d38bf5b329e0fb17c9d08ca6ef61e370f62fab405aac
Jul 14 17:27:12.411025 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.410986    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"}
Jul 14 17:27:12.411438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.411075    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"}
Jul 14 17:27:12.411438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.411118    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"}
Jul 14 17:27:12.411438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.411162    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"}
Jul 14 17:27:12.411438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.411198    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"}
Jul 14 17:27:12.412820 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.412794    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b" exitCode=0
Jul 14 17:27:12.412936 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.412831    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"}
Jul 14 17:27:12.412936 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:12.412884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"3d273eb94b58401e3d63d38bf5b329e0fb17c9d08ca6ef61e370f62fab405aac"}
Jul 14 17:27:13.422313 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:13.422212    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s" event={"ID":"2dee9203-7020-4600-a8e6-898ca56d7411","Type":"ContainerStarted","Data":"cbfdf37330a32ce4e70ff78a3c0bab527f4da6f996d6b93c194763e48d0c10e4"}
Jul 14 17:27:13.423730 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:13.423707    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575" event={"ID":"055a5c57-6812-44e4-995c-3d645338393f","Type":"ContainerStarted","Data":"d7ac091bdc8c48b59541b6318e5dfaf3aa75fc9b31a1ae5efeb429e2471ee464"}
Jul 14 17:27:13.423916 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:13.423901    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:13.428324 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:13.428306    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575"
Jul 14 17:27:13.439029 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:13.438989    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s" podStartSLOduration=1.6899916419999998 podStartE2EDuration="4.43897128s" totalImagesPullingTime="2.748979638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:13.437586104 +0000 UTC m=+79.981989898" watchObservedRunningTime="2026-07-14 17:27:13.43897128 +0000 UTC m=+79.983375074"
Jul 14 17:27:14.432602 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:14.432562    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerStarted","Data":"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"}
Jul 14 17:27:14.458517 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:14.458471    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-r7575" podStartSLOduration=2.681671424 podStartE2EDuration="4.458453684s" totalImagesPullingTime="1.77678226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:13.452380751 +0000 UTC m=+79.996784549" watchObservedRunningTime="2026-07-14 17:27:14.458453684 +0000 UTC m=+81.002857477"
Jul 14 17:27:14.459655 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:14.459605    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.169445647 podStartE2EDuration="8.459594016s" totalImagesPullingTime="5.27138665s" totalInitContainerRuntime="18.761719ms" isStatefulPod=false podCreationTimestamp="2026-07-14 17:27:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:14.457747172 +0000 UTC m=+81.002150965" watchObservedRunningTime="2026-07-14 17:27:14.459594016 +0000 UTC m=+81.003997811"
Jul 14 17:27:16.443786 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:16.443698    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"}
Jul 14 17:27:16.444138 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:16.443789    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"}
Jul 14 17:27:18.453905 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:18.453870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"}
Jul 14 17:27:18.454453 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:18.453949    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"}
Jul 14 17:27:18.454453 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:18.453995    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"}
Jul 14 17:27:18.454453 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:18.454033    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerStarted","Data":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"}
Jul 14 17:27:18.490020 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:18.489963    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.959150848 podStartE2EDuration="7.489943357s" totalImagesPullingTime="5.505499548s" totalInitContainerRuntime="25.292961ms" isStatefulPod=false podCreationTimestamp="2026-07-14 17:27:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:27:18.488123541 +0000 UTC m=+85.032527357" watchObservedRunningTime="2026-07-14 17:27:18.489943357 +0000 UTC m=+85.034347152"
Jul 14 17:27:22.057088 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:22.057040    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:27:30.260602 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:30.260571    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:30.260976 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:30.260648    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:33.353858 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:33.353832    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7knwp"
Jul 14 17:27:50.266618 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:50.266589    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:27:50.270446 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:27:50.270423    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68cc46949c-2kn2s"
Jul 14 17:28:12.057271 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:12.057238    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:12.075233 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:12.075208    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:12.668498 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:12.668471    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:25.792264 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792228    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 17:28:25.792751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792502    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="alertmanager" containerID="cri-o://c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838" gracePeriod=120
Jul 14 17:28:25.792751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792578    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-web" containerID="cri-o://306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a" gracePeriod=120
Jul 14 17:28:25.792751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792578    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-metric" containerID="cri-o://a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab" gracePeriod=120
Jul 14 17:28:25.792751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792621    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="config-reloader" containerID="cri-o://84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4" gracePeriod=120
Jul 14 17:28:25.792751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792669    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="prom-label-proxy" containerID="cri-o://71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce" gracePeriod=120
Jul 14 17:28:25.792751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:25.792676    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy" containerID="cri-o://81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61" gracePeriod=120
Jul 14 17:28:26.702959 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.702926    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce" exitCode=0
Jul 14 17:28:26.702959 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.702953    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61" exitCode=0
Jul 14 17:28:26.702959 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.702961    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4" exitCode=0
Jul 14 17:28:26.702959 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.702968    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838" exitCode=0
Jul 14 17:28:26.703221 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.702998    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"}
Jul 14 17:28:26.703221 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.703060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"}
Jul 14 17:28:26.703221 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.703086    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"}
Jul 14 17:28:26.703221 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:26.703111    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"}
Jul 14 17:28:27.042290 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.042264    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:28:27.067296 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067270    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-web-config\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067327    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-cluster-tls-config\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067396    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067427    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067453    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-main-db\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067476    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-metrics-client-ca\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067505    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067541    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067555    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-config-volume\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067575    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-config-out\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067599    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-tls-assets\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067657    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fg4w\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-kube-api-access-2fg4w\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.067741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.067688    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"f006b60b-4919-4801-8163-1ff15765dc1e\" (UID: \"f006b60b-4919-4801-8163-1ff15765dc1e\") "
Jul 14 17:28:27.070645 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.070322    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:27.070645 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.070593    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:28:27.072108 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.072081    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:27.072521 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.072495    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-config-volume" (OuterVolumeSpecName: "config-volume") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.072617 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.072518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-config-out" (OuterVolumeSpecName: "config-out") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:28:27.074666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.074609    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:28:27.074666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.074614    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.074829 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.074804    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.075891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.075852    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.075985 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.075920    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.076940 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.076832    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-kube-api-access-2fg4w" (OuterVolumeSpecName: "kube-api-access-2fg4w") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "kube-api-access-2fg4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:28:27.080978 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.080947    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.085625 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.085599    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-web-config" (OuterVolumeSpecName: "web-config") pod "f006b60b-4919-4801-8163-1ff15765dc1e" (UID: "f006b60b-4919-4801-8163-1ff15765dc1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:27.169173 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169131    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-metrics-client-ca\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169173 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169170    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-main-tls\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169173 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169188    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169199    2567 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-config-volume\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169208    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-config-out\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169215    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-tls-assets\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169223    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fg4w\" (UniqueName: \"kubernetes.io/projected/f006b60b-4919-4801-8163-1ff15765dc1e-kube-api-access-2fg4w\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169231    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169240    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-web-config\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169247    2567 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-cluster-tls-config\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169255    2567 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f006b60b-4919-4801-8163-1ff15765dc1e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169263    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.169403 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.169271    2567 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f006b60b-4919-4801-8163-1ff15765dc1e-alertmanager-main-db\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:27.709682 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709648    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab" exitCode=0
Jul 14 17:28:27.709682 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709673    2567 generic.go:356] "Generic (PLEG): container finished" podID="f006b60b-4919-4801-8163-1ff15765dc1e" containerID="306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a" exitCode=0
Jul 14 17:28:27.709891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709734    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 17:28:27.709891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709733    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"}
Jul 14 17:28:27.709891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709806    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"}
Jul 14 17:28:27.709891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709834    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f006b60b-4919-4801-8163-1ff15765dc1e","Type":"ContainerDied","Data":"273df5a8797897b7c4a9fafe6fa0f4f3ddb3cc6367abd474ed641d6b84a8ceb7"}
Jul 14 17:28:27.709891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.709869    2567 scope.go:117] "RemoveContainer" containerID="71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"
Jul 14 17:28:27.717784 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.717765    2567 scope.go:117] "RemoveContainer" containerID="a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"
Jul 14 17:28:27.724070 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.724053    2567 scope.go:117] "RemoveContainer" containerID="81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"
Jul 14 17:28:27.730588 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.730563    2567 scope.go:117] "RemoveContainer" containerID="306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"
Jul 14 17:28:27.732536 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.732513    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 17:28:27.736286 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.736266    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 17:28:27.738225 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.738212    2567 scope.go:117] "RemoveContainer" containerID="84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"
Jul 14 17:28:27.744675 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.744661    2567 scope.go:117] "RemoveContainer" containerID="c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"
Jul 14 17:28:27.751063 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.751048    2567 scope.go:117] "RemoveContainer" containerID="09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"
Jul 14 17:28:27.757759 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.757736    2567 scope.go:117] "RemoveContainer" containerID="71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"
Jul 14 17:28:27.758035 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.758016    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce\": container with ID starting with 71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce not found: ID does not exist" containerID="71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"
Jul 14 17:28:27.758080 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758042    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"} err="failed to get container status \"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce\": rpc error: code = NotFound desc = could not find container \"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce\": container with ID starting with 71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce not found: ID does not exist"
Jul 14 17:28:27.758123 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758082    2567 scope.go:117] "RemoveContainer" containerID="a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"
Jul 14 17:28:27.758291 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.758276    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab\": container with ID starting with a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab not found: ID does not exist" containerID="a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"
Jul 14 17:28:27.758333 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758298    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"} err="failed to get container status \"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab\": rpc error: code = NotFound desc = could not find container \"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab\": container with ID starting with a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab not found: ID does not exist"
Jul 14 17:28:27.758333 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758311    2567 scope.go:117] "RemoveContainer" containerID="81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"
Jul 14 17:28:27.758517 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.758503    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61\": container with ID starting with 81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61 not found: ID does not exist" containerID="81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"
Jul 14 17:28:27.758551 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758520    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"} err="failed to get container status \"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61\": rpc error: code = NotFound desc = could not find container \"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61\": container with ID starting with 81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61 not found: ID does not exist"
Jul 14 17:28:27.758551 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758532    2567 scope.go:117] "RemoveContainer" containerID="306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"
Jul 14 17:28:27.758728 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.758713    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a\": container with ID starting with 306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a not found: ID does not exist" containerID="306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"
Jul 14 17:28:27.758773 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758730    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"} err="failed to get container status \"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a\": rpc error: code = NotFound desc = could not find container \"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a\": container with ID starting with 306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a not found: ID does not exist"
Jul 14 17:28:27.758773 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758743    2567 scope.go:117] "RemoveContainer" containerID="84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"
Jul 14 17:28:27.758938 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.758921    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4\": container with ID starting with 84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4 not found: ID does not exist" containerID="84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"
Jul 14 17:28:27.758995 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758948    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"} err="failed to get container status \"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4\": rpc error: code = NotFound desc = could not find container \"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4\": container with ID starting with 84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4 not found: ID does not exist"
Jul 14 17:28:27.758995 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.758971    2567 scope.go:117] "RemoveContainer" containerID="c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"
Jul 14 17:28:27.759197 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.759175    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838\": container with ID starting with c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838 not found: ID does not exist" containerID="c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"
Jul 14 17:28:27.759263 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.759205    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"} err="failed to get container status \"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838\": rpc error: code = NotFound desc = could not find container \"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838\": container with ID starting with c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838 not found: ID does not exist"
Jul 14 17:28:27.759263 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.759219    2567 scope.go:117] "RemoveContainer" containerID="09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"
Jul 14 17:28:27.759581 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:27.759556    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60\": container with ID starting with 09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60 not found: ID does not exist" containerID="09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"
Jul 14 17:28:27.759677 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.759588    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"} err="failed to get container status \"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60\": rpc error: code = NotFound desc = could not find container \"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60\": container with ID starting with 09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60 not found: ID does not exist"
Jul 14 17:28:27.759677 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.759608    2567 scope.go:117] "RemoveContainer" containerID="71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"
Jul 14 17:28:27.759923 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.759893    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce"} err="failed to get container status \"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce\": rpc error: code = NotFound desc = could not find container \"71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce\": container with ID starting with 71b7467d890ed69f281cf221010a30b794bb03cd60f6e8abc6e394376098f4ce not found: ID does not exist"
Jul 14 17:28:27.759923 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.759917    2567 scope.go:117] "RemoveContainer" containerID="a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"
Jul 14 17:28:27.760168 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760148    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab"} err="failed to get container status \"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab\": rpc error: code = NotFound desc = could not find container \"a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab\": container with ID starting with a486b87803db97b5bb32e600dccaa128c1cd64fe248bb109cfe7277ad6127cab not found: ID does not exist"
Jul 14 17:28:27.760216 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760169    2567 scope.go:117] "RemoveContainer" containerID="81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"
Jul 14 17:28:27.760461 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760439    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61"} err="failed to get container status \"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61\": rpc error: code = NotFound desc = could not find container \"81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61\": container with ID starting with 81aa0bd7e6bfeccc278150580af914ca267b84894a42889c36af20770b882d61 not found: ID does not exist"
Jul 14 17:28:27.760538 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760462    2567 scope.go:117] "RemoveContainer" containerID="306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"
Jul 14 17:28:27.760691 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760672    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a"} err="failed to get container status \"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a\": rpc error: code = NotFound desc = could not find container \"306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a\": container with ID starting with 306b4213c20322398a7f559bc01f0b35899fd54f64fa1f0ec8bb873bcb0c7d4a not found: ID does not exist"
Jul 14 17:28:27.760738 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760691    2567 scope.go:117] "RemoveContainer" containerID="84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"
Jul 14 17:28:27.760889 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760872    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4"} err="failed to get container status \"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4\": rpc error: code = NotFound desc = could not find container \"84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4\": container with ID starting with 84d7fcfe834c42e445356bb77b1c26ac73bc4be38c0d14663abb5de7df3227c4 not found: ID does not exist"
Jul 14 17:28:27.760930 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.760890    2567 scope.go:117] "RemoveContainer" containerID="c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"
Jul 14 17:28:27.761091 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.761075    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838"} err="failed to get container status \"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838\": rpc error: code = NotFound desc = could not find container \"c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838\": container with ID starting with c51e61cad6eb718777e11309110c608787ebb35e5ad4477a08be9974cc541838 not found: ID does not exist"
Jul 14 17:28:27.761134 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.761093    2567 scope.go:117] "RemoveContainer" containerID="09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"
Jul 14 17:28:27.761301 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:27.761285    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60"} err="failed to get container status \"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60\": rpc error: code = NotFound desc = could not find container \"09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60\": container with ID starting with 09f4eb7351b349f5cc4a7c612d86451cd6218ad1f8e759fb4892a9cfa7347a60 not found: ID does not exist"
Jul 14 17:28:28.050398 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:28.050297    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" path="/var/lib/kubelet/pods/f006b60b-4919-4801-8163-1ff15765dc1e/volumes"
Jul 14 17:28:29.828041 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828004    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"]
Jul 14 17:28:29.828506 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828423    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-metric"
Jul 14 17:28:29.828506 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828441    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-metric"
Jul 14 17:28:29.828506 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828460    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="init-config-reloader"
Jul 14 17:28:29.828506 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828468    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="init-config-reloader"
Jul 14 17:28:29.828506 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828477    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-web"
Jul 14 17:28:29.828506 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828486    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-web"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828511    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="config-reloader"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828520    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="config-reloader"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828528    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="prom-label-proxy"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828536    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="prom-label-proxy"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828547    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="alertmanager"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828555    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="alertmanager"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828564    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828572    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828634    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="prom-label-proxy"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828647    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="alertmanager"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828657    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="config-reloader"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828668    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-web"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828676    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy"
Jul 14 17:28:29.828802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.828685    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f006b60b-4919-4801-8163-1ff15765dc1e" containerName="kube-rbac-proxy-metric"
Jul 14 17:28:29.833722 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.833701    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.836535 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.836513    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 14 17:28:29.836663 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.836580    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 14 17:28:29.836663 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.836586    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 14 17:28:29.836789 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.836767    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 14 17:28:29.836906 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.836892    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 14 17:28:29.837032 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.837007    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-89nh8\""
Jul 14 17:28:29.841535 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.841516    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 14 17:28:29.844370 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.844335    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"]
Jul 14 17:28:29.890575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-secret-telemeter-client\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890579    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-serving-certs-ca-bundle\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ngk\" (UniqueName: \"kubernetes.io/projected/910ebf57-1b42-4a8d-9170-0ee64f399cd1-kube-api-access-j9ngk\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890792 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890682    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890792 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-telemeter-client-tls\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890792 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890742    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-federate-client-tls\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.890892 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.890804    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-metrics-client-ca\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.991960 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.991933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992065 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.991976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-telemeter-client-tls\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992065 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-federate-client-tls\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992065 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992030    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-metrics-client-ca\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992170 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-secret-telemeter-client\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992170 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-serving-certs-ca-bundle\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992170 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992145    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992293 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ngk\" (UniqueName: \"kubernetes.io/projected/910ebf57-1b42-4a8d-9170-0ee64f399cd1-kube-api-access-j9ngk\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.992911 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-metrics-client-ca\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.993018 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992945    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.993018 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.992957    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/910ebf57-1b42-4a8d-9170-0ee64f399cd1-serving-certs-ca-bundle\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.994586 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.994564    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-secret-telemeter-client\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.994675 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.994658    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.994715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.994665    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-telemeter-client-tls\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:29.994715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:29.994702    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/910ebf57-1b42-4a8d-9170-0ee64f399cd1-federate-client-tls\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:30.000609 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.000589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ngk\" (UniqueName: \"kubernetes.io/projected/910ebf57-1b42-4a8d-9170-0ee64f399cd1-kube-api-access-j9ngk\") pod \"telemeter-client-78d46c5b8c-4scmj\" (UID: \"910ebf57-1b42-4a8d-9170-0ee64f399cd1\") " pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:30.098112 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098036    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:28:30.098504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098377    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="thanos-sidecar" containerID="cri-o://828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56" gracePeriod=600
Jul 14 17:28:30.098504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098377    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="prometheus" containerID="cri-o://1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938" gracePeriod=600
Jul 14 17:28:30.098504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098428    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="config-reloader" containerID="cri-o://e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad" gracePeriod=600
Jul 14 17:28:30.098504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098435    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-web" containerID="cri-o://4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc" gracePeriod=600
Jul 14 17:28:30.098504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098383    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy" containerID="cri-o://9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46" gracePeriod=600
Jul 14 17:28:30.098504 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.098427    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42" gracePeriod=600
Jul 14 17:28:30.143488 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.143466    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"
Jul 14 17:28:30.276277 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.276252    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d46c5b8c-4scmj"]
Jul 14 17:28:30.281494 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:28:30.281469    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910ebf57_1b42_4a8d_9170_0ee64f399cd1.slice/crio-e187563793d08dec3483053a951fb24a6e1e63fd4b31bee5d50c2d7a4c537c97 WatchSource:0}: Error finding container e187563793d08dec3483053a951fb24a6e1e63fd4b31bee5d50c2d7a4c537c97: Status 404 returned error can't find the container with id e187563793d08dec3483053a951fb24a6e1e63fd4b31bee5d50c2d7a4c537c97
Jul 14 17:28:30.335092 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.335069    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.394969 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.394891    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-thanos-prometheus-http-client-file\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.394969 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.394933    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-0\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.394969 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.394966    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-tls\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395231 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.394981    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-grpc-tls\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395231 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395091    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-kubelet-serving-ca-bundle\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395231 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395142    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-db\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395231 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395170    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-1\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395231 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395198    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-metrics-client-ca\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395231 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395221    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-2\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395251    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395284    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-config-out\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395315    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-web-config\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395340    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-trusted-ca-bundle\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395405    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59j5w\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-kube-api-access-59j5w\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395430    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-config\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395493    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-serving-certs-ca-bundle\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395523    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-kube-rbac-proxy\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395549    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395958 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395584    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-tls-assets\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395958 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395609    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-metrics-client-certs\") pod \"93e4e93c-3103-4538-b713-dff5ec8228f8\" (UID: \"93e4e93c-3103-4538-b713-dff5ec8228f8\") "
Jul 14 17:28:30.395958 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395639    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.395958 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.395849    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.396158 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.396070    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.396714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.396365    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.396714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.396453    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:28:30.396866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.396596    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.396866 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.396821    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.397276 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.397085    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.397603 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.397491    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.397702 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.397678    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:28:30.397858 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.397835    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.399335 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.399304    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.400105 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400074    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:28:30.400259 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400237    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-config" (OuterVolumeSpecName: "config") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.400363 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400243    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-kube-api-access-59j5w" (OuterVolumeSpecName: "kube-api-access-59j5w") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "kube-api-access-59j5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:28:30.400363 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400258    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.400363 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400328    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-config-out" (OuterVolumeSpecName: "config-out") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:28:30.400495 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400360    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.400791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.400768    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.401501 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.401480    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.409899 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.409873    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-web-config" (OuterVolumeSpecName: "web-config") pod "93e4e93c-3103-4538-b713-dff5ec8228f8" (UID: "93e4e93c-3103-4538-b713-dff5ec8228f8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:28:30.497138 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497106    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497138 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497133    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497138 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497143    2567 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-grpc-tls\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497151    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-db\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497159    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497167    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-metrics-client-ca\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497176    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497197    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497205    2567 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93e4e93c-3103-4538-b713-dff5ec8228f8-config-out\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497213    2567 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-web-config\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497220    2567 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497228    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59j5w\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-kube-api-access-59j5w\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497235    2567 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-config\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497243    2567 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93e4e93c-3103-4538-b713-dff5ec8228f8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497251    2567 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-kube-rbac-proxy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497259    2567 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497268    2567 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93e4e93c-3103-4538-b713-dff5ec8228f8-tls-assets\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497276    2567 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-secret-metrics-client-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.497373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.497283    2567 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/93e4e93c-3103-4538-b713-dff5ec8228f8-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:28:30.723271 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.723181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj" event={"ID":"910ebf57-1b42-4a8d-9170-0ee64f399cd1","Type":"ContainerStarted","Data":"e187563793d08dec3483053a951fb24a6e1e63fd4b31bee5d50c2d7a4c537c97"}
Jul 14 17:28:30.726178 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726155    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42" exitCode=0
Jul 14 17:28:30.726178 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726176    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46" exitCode=0
Jul 14 17:28:30.726178 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726182    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc" exitCode=0
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726188    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56" exitCode=0
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726193    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad" exitCode=0
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726199    2567 generic.go:356] "Generic (PLEG): container finished" podID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938" exitCode=0
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726214    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"}
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726227    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726261    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"}
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"}
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726338    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"}
Jul 14 17:28:30.726365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726359    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.726684 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726388    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"}
Jul 14 17:28:30.726684 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726424    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"}
Jul 14 17:28:30.726684 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.726447    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"93e4e93c-3103-4538-b713-dff5ec8228f8","Type":"ContainerDied","Data":"3d273eb94b58401e3d63d38bf5b329e0fb17c9d08ca6ef61e370f62fab405aac"}
Jul 14 17:28:30.734420 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.734399    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.741601 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.741584    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.747880 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.747865    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.751656 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.751636    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:28:30.754988 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.754961    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.756576 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.756559    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:28:30.761230 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.761214    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.767578 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.767560    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.773500 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.773481    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.773857 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.773839    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.773924 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.773863    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"} err="failed to get container status \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist"
Jul 14 17:28:30.773924 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.773879    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.774088 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.774072    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.774127 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774092    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"} err="failed to get container status \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist"
Jul 14 17:28:30.774127 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774105    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.774302 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.774267    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.774302 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774288    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"} err="failed to get container status \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist"
Jul 14 17:28:30.774302 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774299    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.774617 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.774583    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.774652 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774637    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"} err="failed to get container status \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist"
Jul 14 17:28:30.774684 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774652    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.774873 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.774859    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.774910 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774877    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"} err="failed to get container status \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist"
Jul 14 17:28:30.774910 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.774889    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.775112 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.775095    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.775156 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775117    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"} err="failed to get container status \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist"
Jul 14 17:28:30.775156 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775131    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.775398 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:28:30.775373    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.775478 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775401    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"} err="failed to get container status \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist"
Jul 14 17:28:30.775478 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775415    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.775608 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775592    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"} err="failed to get container status \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist"
Jul 14 17:28:30.775647 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775609    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.775810 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775795    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"} err="failed to get container status \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist"
Jul 14 17:28:30.775810 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.775810    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.776025 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776010    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"} err="failed to get container status \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist"
Jul 14 17:28:30.776025 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776024    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.776200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776183    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"} err="failed to get container status \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist"
Jul 14 17:28:30.776237 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776201    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.776412 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776392    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"} err="failed to get container status \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist"
Jul 14 17:28:30.776492 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776414    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.776597 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776583    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"} err="failed to get container status \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist"
Jul 14 17:28:30.776641 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776598    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.776803 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776780    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"} err="failed to get container status \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist"
Jul 14 17:28:30.776803 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776804    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.776988 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776971    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"} err="failed to get container status \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist"
Jul 14 17:28:30.777032 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.776989    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.777200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777177    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"} err="failed to get container status \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist"
Jul 14 17:28:30.777257 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777200    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.777419 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777404    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"} err="failed to get container status \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist"
Jul 14 17:28:30.777465 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777420    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.777587 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777572    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"} err="failed to get container status \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist"
Jul 14 17:28:30.777626 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777587    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.777780 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777766    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"} err="failed to get container status \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist"
Jul 14 17:28:30.777817 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777780    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.777945 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777931    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"} err="failed to get container status \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist"
Jul 14 17:28:30.777983 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.777946    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.778113 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778099    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"} err="failed to get container status \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist"
Jul 14 17:28:30.778147 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778114    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.778303 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778289    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"} err="failed to get container status \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist"
Jul 14 17:28:30.778339 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778302    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.778515 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778492    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"} err="failed to get container status \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist"
Jul 14 17:28:30.778560 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778517    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.778736 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778719    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"} err="failed to get container status \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist"
Jul 14 17:28:30.778786 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778735    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.778911 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778892    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"} err="failed to get container status \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist"
Jul 14 17:28:30.778948 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.778912    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.779131 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779114    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"} err="failed to get container status \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist"
Jul 14 17:28:30.779131 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779130    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.779359 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779323    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"} err="failed to get container status \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist"
Jul 14 17:28:30.779405 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779359    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.779546 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779531    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"} err="failed to get container status \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist"
Jul 14 17:28:30.779582 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779547    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.779735 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779718    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"} err="failed to get container status \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist"
Jul 14 17:28:30.779784 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779734    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.779900 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779885    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"} err="failed to get container status \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist"
Jul 14 17:28:30.779933 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.779907    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.780109 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780092    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"} err="failed to get container status \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist"
Jul 14 17:28:30.780148 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780109    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.780294 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780271    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"} err="failed to get container status \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist"
Jul 14 17:28:30.780294 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780287    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.780468 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780452    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"} err="failed to get container status \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist"
Jul 14 17:28:30.780514 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780469    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.780650 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780635    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"} err="failed to get container status \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist"
Jul 14 17:28:30.780699 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780650    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.780814 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780800    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"} err="failed to get container status \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist"
Jul 14 17:28:30.780852 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.780814    2567 scope.go:117] "RemoveContainer" containerID="a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"
Jul 14 17:28:30.781371 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.781289    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42"} err="failed to get container status \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": rpc error: code = NotFound desc = could not find container \"a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42\": container with ID starting with a2c4a4132dd22b327aef46df06ab517756413fe5442c644a74ee867852b9aa42 not found: ID does not exist"
Jul 14 17:28:30.781463 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.781400    2567 scope.go:117] "RemoveContainer" containerID="9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"
Jul 14 17:28:30.781711 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.781676    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46"} err="failed to get container status \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": rpc error: code = NotFound desc = could not find container \"9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46\": container with ID starting with 9a26d83bfdc9cca6507a917646cbc9fbba4517e4fcf0c941cb091a57ebb8bc46 not found: ID does not exist"
Jul 14 17:28:30.781789 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.781712    2567 scope.go:117] "RemoveContainer" containerID="4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"
Jul 14 17:28:30.781974 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.781956    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc"} err="failed to get container status \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": rpc error: code = NotFound desc = could not find container \"4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc\": container with ID starting with 4cc339a08aea478709e07917e5b7766d2ba51cfd962cf526dbceea85581757dc not found: ID does not exist"
Jul 14 17:28:30.781974 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.781974    2567 scope.go:117] "RemoveContainer" containerID="828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"
Jul 14 17:28:30.782187 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782171    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56"} err="failed to get container status \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": rpc error: code = NotFound desc = could not find container \"828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56\": container with ID starting with 828d22b5a707cf42c00dc9d7d7e13146da3fa4ca1b3032789832278518497e56 not found: ID does not exist"
Jul 14 17:28:30.782254 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782188    2567 scope.go:117] "RemoveContainer" containerID="e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"
Jul 14 17:28:30.782418 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782398    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad"} err="failed to get container status \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": rpc error: code = NotFound desc = could not find container \"e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad\": container with ID starting with e79cf61eb5f41e09023577edc88f6f900adf8c996177cd1ae0ba4613d631b2ad not found: ID does not exist"
Jul 14 17:28:30.782418 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782418    2567 scope.go:117] "RemoveContainer" containerID="1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"
Jul 14 17:28:30.782622 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782607    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938"} err="failed to get container status \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": rpc error: code = NotFound desc = could not find container \"1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938\": container with ID starting with 1bbd3b6e775eb23f524bdff220b0baccafe12e0c63da0f0ce7de0d1bd1d92938 not found: ID does not exist"
Jul 14 17:28:30.782622 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782622    2567 scope.go:117] "RemoveContainer" containerID="469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"
Jul 14 17:28:30.782808 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.782793    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b"} err="failed to get container status \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": rpc error: code = NotFound desc = could not find container \"469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b\": container with ID starting with 469f04799705d87e281e7b96d90f2ad99b216e865dc335754aabd0e7bee50c5b not found: ID does not exist"
Jul 14 17:28:30.783228 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783212    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:28:30.783497 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783482    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="config-reloader"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783499    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="config-reloader"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783514    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="prometheus"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783521    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="prometheus"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783531    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783539    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783551    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="init-config-reloader"
Jul 14 17:28:30.783575 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783559    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="init-config-reloader"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783583    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="thanos-sidecar"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783591    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="thanos-sidecar"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783600    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-web"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783608    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-web"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783618    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-thanos"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783626    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-thanos"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783686    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="thanos-sidecar"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783697    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="prometheus"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783705    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-thanos"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783713    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="config-reloader"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783723    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy-web"
Jul 14 17:28:30.783886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.783731    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" containerName="kube-rbac-proxy"
Jul 14 17:28:30.788939 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.788922    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.791691 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.791674    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 17:28:30.791780 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.791762    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 17:28:30.792133 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.792118    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-btvtb\""
Jul 14 17:28:30.792212 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.792197    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 17:28:30.792656 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.792636    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 17:28:30.793662 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793039    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3bkh8es2oqpa0\""
Jul 14 17:28:30.793662 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793092    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 17:28:30.793662 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793044    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 17:28:30.793662 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793401    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 17:28:30.793662 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793435    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 17:28:30.795126 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793688    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 17:28:30.795126 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793774    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 17:28:30.795126 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.793853    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 17:28:30.795126 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.794463    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 17:28:30.799325 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.799309    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 17:28:30.800935 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.800888    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 17:28:30.803109 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.803092    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:28:30.899531 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899499    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-config\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899531 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899557    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899575    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899617    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nq8\" (UniqueName: \"kubernetes.io/projected/6583dc36-edf9-4496-9e33-c21d01944044-kube-api-access-45nq8\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6583dc36-edf9-4496-9e33-c21d01944044-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899698    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899730    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6583dc36-edf9-4496-9e33-c21d01944044-config-out\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899745    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899790    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899803    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899829    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.899932 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899845    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.900323 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899860    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.900323 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899878    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-web-config\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.900323 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899893    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.900323 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899908    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:30.900323 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:30.899945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000317 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6583dc36-edf9-4496-9e33-c21d01944044-config-out\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000317 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000255    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000317 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000317 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000290    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000317 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000303    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-web-config\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000427    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000448    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000470    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-config\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000532    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000585    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000605    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nq8\" (UniqueName: \"kubernetes.io/projected/6583dc36-edf9-4496-9e33-c21d01944044-kube-api-access-45nq8\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.000643 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000636    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.002097 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000664    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6583dc36-edf9-4496-9e33-c21d01944044-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.002097 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.002097 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.000781    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.003423 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.003297    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.003423 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.003303    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6583dc36-edf9-4496-9e33-c21d01944044-config-out\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.003423 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.003416    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-config\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.004182 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.004095    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.004182 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.004132    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.004557 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.004509    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.004844 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.004808    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.005202 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.005175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.005929 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.005677    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.006130 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.006092    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.006468 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.006421    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.006665 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.006644    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.006886 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.006842    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.007009 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.006960    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6583dc36-edf9-4496-9e33-c21d01944044-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.007152 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.007124    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6583dc36-edf9-4496-9e33-c21d01944044-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.008388 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.008362    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.008541 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.008443    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-web-config\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.008599 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.008586    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6583dc36-edf9-4496-9e33-c21d01944044-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.034361 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.033657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nq8\" (UniqueName: \"kubernetes.io/projected/6583dc36-edf9-4496-9e33-c21d01944044-kube-api-access-45nq8\") pod \"prometheus-k8s-0\" (UID: \"6583dc36-edf9-4496-9e33-c21d01944044\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.100982 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.100947    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:28:31.232916 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.232893    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 17:28:31.235064 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:28:31.235039    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6583dc36_edf9_4496_9e33_c21d01944044.slice/crio-a8383bbdde258fc3ef1d053bf08ef09ad875a9ca5c13cac343a4d96dfdd94741 WatchSource:0}: Error finding container a8383bbdde258fc3ef1d053bf08ef09ad875a9ca5c13cac343a4d96dfdd94741: Status 404 returned error can't find the container with id a8383bbdde258fc3ef1d053bf08ef09ad875a9ca5c13cac343a4d96dfdd94741
Jul 14 17:28:31.733005 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.732973    2567 generic.go:356] "Generic (PLEG): container finished" podID="6583dc36-edf9-4496-9e33-c21d01944044" containerID="9de7b32f2d6771309d1457e22da74714ce67100e67faea2f8ed4c1878c240f4e" exitCode=0
Jul 14 17:28:31.733156 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.733012    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerDied","Data":"9de7b32f2d6771309d1457e22da74714ce67100e67faea2f8ed4c1878c240f4e"}
Jul 14 17:28:31.733156 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:31.733060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"a8383bbdde258fc3ef1d053bf08ef09ad875a9ca5c13cac343a4d96dfdd94741"}
Jul 14 17:28:32.051698 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.051677    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e4e93c-3103-4538-b713-dff5ec8228f8" path="/var/lib/kubelet/pods/93e4e93c-3103-4538-b713-dff5ec8228f8/volumes"
Jul 14 17:28:32.741951 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.741909    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"8117d99b80760975911ef8a669b03fd665a8764956302a23a73459568abb6ce6"}
Jul 14 17:28:32.742131 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.742041    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"744365a0e510df81e99e6effbfd5780c9939444d7d6a89fcd03559dbbe0531aa"}
Jul 14 17:28:32.742131 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.742087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"e9a9d119d57f998e686b0688d25990de328469df66b43357cbf3917b96bb8054"}
Jul 14 17:28:32.742131 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.742128    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"94ca76b9108fb7ccce5c1f59d395e471e5ab784d706a0ebd163a2aa021ee4d41"}
Jul 14 17:28:32.742285 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.742166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"822566bb35e2016c5b3dcc0e3c9095573496eaef1108d536a0f9cad01a607fe6"}
Jul 14 17:28:32.742285 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.742201    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6583dc36-edf9-4496-9e33-c21d01944044","Type":"ContainerStarted","Data":"8b1c4baee04ae541a46bae276beb05fccc6bb16202785caafac2e55bd4ec7dd2"}
Jul 14 17:28:32.770595 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:32.770542    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.752417863 podStartE2EDuration="2.770526212s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.108349ms" isStatefulPod=false podCreationTimestamp="2026-07-14 17:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:28:32.768512801 +0000 UTC m=+159.312916594" watchObservedRunningTime="2026-07-14 17:28:32.770526212 +0000 UTC m=+159.314930006"
Jul 14 17:28:33.748335 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:33.748240    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj" event={"ID":"910ebf57-1b42-4a8d-9170-0ee64f399cd1","Type":"ContainerStarted","Data":"d8176995cd44d74301860c8ebcc302114a1e9c05070748be886858d593533a07"}
Jul 14 17:28:33.748335 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:33.748299    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj" event={"ID":"910ebf57-1b42-4a8d-9170-0ee64f399cd1","Type":"ContainerStarted","Data":"300923e133c8ba39e81b79fa627ce337a2681c360d5e0d4d9c380fb6d836b3a9"}
Jul 14 17:28:33.748335 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:33.748318    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj" event={"ID":"910ebf57-1b42-4a8d-9170-0ee64f399cd1","Type":"ContainerStarted","Data":"97f369d5aea69450d03274e88514cd9e2bf34f2f3346aa648da4954d1f79fdf6"}
Jul 14 17:28:33.771052 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:33.771004    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78d46c5b8c-4scmj" podStartSLOduration=1.642325875 podStartE2EDuration="4.770989649s" totalImagesPullingTime="3.128663774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:28:33.769738773 +0000 UTC m=+160.314142589" watchObservedRunningTime="2026-07-14 17:28:33.770989649 +0000 UTC m=+160.315393443"
Jul 14 17:28:36.101128 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:28:36.101097    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:29:31.102142 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:29:31.102099    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:29:31.117200 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:29:31.117175    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:29:31.962006 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:29:31.961980    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 17:30:53.796495 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:30:53.796466    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:30:53.802000 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:30:53.801973    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:30:53.802510 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:30:53.802490    2567 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 17:32:01.439844 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.439764    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"]
Jul 14 17:32:01.443089 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.443069    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.446908 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.446884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 17:32:01.447023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.446928    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 17:32:01.447023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.446884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8xbhw\""
Jul 14 17:32:01.447023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.446884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 17:32:01.447023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.446888    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 17:32:01.447023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.446894    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 17:32:01.454229 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.454205    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"]
Jul 14 17:32:01.517085 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.517046    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d84d2662-bafb-4ade-983a-2893c99cf79d-manager-config\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.517251 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.517089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d84d2662-bafb-4ade-983a-2893c99cf79d-metrics-cert\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.517251 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.517175    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69p4b\" (UniqueName: \"kubernetes.io/projected/d84d2662-bafb-4ade-983a-2893c99cf79d-kube-api-access-69p4b\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.517251 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.517208    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84d2662-bafb-4ade-983a-2893c99cf79d-cert\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.618536 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.618496    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d84d2662-bafb-4ade-983a-2893c99cf79d-manager-config\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.618536 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.618534    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d84d2662-bafb-4ade-983a-2893c99cf79d-metrics-cert\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.618753 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.618582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69p4b\" (UniqueName: \"kubernetes.io/projected/d84d2662-bafb-4ade-983a-2893c99cf79d-kube-api-access-69p4b\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.618753 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.618611    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84d2662-bafb-4ade-983a-2893c99cf79d-cert\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.619190 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.619170    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d84d2662-bafb-4ade-983a-2893c99cf79d-manager-config\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.621023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.621000    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d84d2662-bafb-4ade-983a-2893c99cf79d-metrics-cert\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.621125 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.621040    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d84d2662-bafb-4ade-983a-2893c99cf79d-cert\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.633933 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.633911    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69p4b\" (UniqueName: \"kubernetes.io/projected/d84d2662-bafb-4ade-983a-2893c99cf79d-kube-api-access-69p4b\") pod \"lws-controller-manager-7cb94d9664-qbq9j\" (UID: \"d84d2662-bafb-4ade-983a-2893c99cf79d\") " pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.752333 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.752256    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:01.904444 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.904421    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"]
Jul 14 17:32:01.906902 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:32:01.906877    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd84d2662_bafb_4ade_983a_2893c99cf79d.slice/crio-7ef6d887419adedb2deea33d03f23c0429cdf929e0e6fef16bc87ffe1eea6843 WatchSource:0}: Error finding container 7ef6d887419adedb2deea33d03f23c0429cdf929e0e6fef16bc87ffe1eea6843: Status 404 returned error can't find the container with id 7ef6d887419adedb2deea33d03f23c0429cdf929e0e6fef16bc87ffe1eea6843
Jul 14 17:32:01.909272 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:01.909255    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:32:02.479745 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:02.479710    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j" event={"ID":"d84d2662-bafb-4ade-983a-2893c99cf79d","Type":"ContainerStarted","Data":"7ef6d887419adedb2deea33d03f23c0429cdf929e0e6fef16bc87ffe1eea6843"}
Jul 14 17:32:03.870462 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.870425    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"]
Jul 14 17:32:03.879079 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.879048    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:03.881745 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.881722    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 17:32:03.881877 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.881730    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 17:32:03.881877 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.881774    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 17:32:03.881877 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.881784    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4xg4v\""
Jul 14 17:32:03.882204 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.882185    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 17:32:03.893632 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.893607    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"]
Jul 14 17:32:03.938411 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.938373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:03.938590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.938455    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:03.938590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.938505    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-tmp\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:03.938590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.938552    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ghw\" (UniqueName: \"kubernetes.io/projected/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-kube-api-access-m9ghw\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:03.938590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:03.938585    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-manifests\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.039752 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.039715    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.039919 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.039783    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-tmp\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.039919 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.039833    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ghw\" (UniqueName: \"kubernetes.io/projected/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-kube-api-access-m9ghw\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.040042 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.039971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-manifests\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.040097 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.040051    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.040451 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.040404    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-manifests\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.042618 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.042595    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-tmp\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.042740 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.042703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.042802 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.042757    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.048365 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.048293    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ghw\" (UniqueName: \"kubernetes.io/projected/1dcf83b9-cc2d-466b-be3d-f6bfe1757234-kube-api-access-m9ghw\") pod \"opendatahub-operator-controller-manager-7f54d68b84-glz67\" (UID: \"1dcf83b9-cc2d-466b-be3d-f6bfe1757234\") " pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.191337 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.191255    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:04.620213 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:04.620183    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"]
Jul 14 17:32:04.622624 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:32:04.622602    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcf83b9_cc2d_466b_be3d_f6bfe1757234.slice/crio-cd298ce88ca1e3d03bbe4c01efe2ca4a4d8daba92fa193465f9c6f0f996ceade WatchSource:0}: Error finding container cd298ce88ca1e3d03bbe4c01efe2ca4a4d8daba92fa193465f9c6f0f996ceade: Status 404 returned error can't find the container with id cd298ce88ca1e3d03bbe4c01efe2ca4a4d8daba92fa193465f9c6f0f996ceade
Jul 14 17:32:05.494658 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:05.494587    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67" event={"ID":"1dcf83b9-cc2d-466b-be3d-f6bfe1757234","Type":"ContainerStarted","Data":"cd298ce88ca1e3d03bbe4c01efe2ca4a4d8daba92fa193465f9c6f0f996ceade"}
Jul 14 17:32:05.496295 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:05.496264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j" event={"ID":"d84d2662-bafb-4ade-983a-2893c99cf79d","Type":"ContainerStarted","Data":"e74cee87b68b1a78c949c6649742be91a168568e3ade1716c578bec30b305941"}
Jul 14 17:32:05.496462 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:05.496413    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:05.518752 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:05.518704    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j" podStartSLOduration=1.876064993 podStartE2EDuration="4.518689711s" totalImagesPullingTime="2.642624718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:05.517399795 +0000 UTC m=+372.061803590" watchObservedRunningTime="2026-07-14 17:32:05.518689711 +0000 UTC m=+372.063093505"
Jul 14 17:32:07.505935 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:07.505906    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67" event={"ID":"1dcf83b9-cc2d-466b-be3d-f6bfe1757234","Type":"ContainerStarted","Data":"8ab664190628846cdec5aac118f8c627c50d24954496d2a67530a671045af488"}
Jul 14 17:32:09.514608 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:09.514574    2567 generic.go:356] "Generic (PLEG): container finished" podID="1dcf83b9-cc2d-466b-be3d-f6bfe1757234" containerID="8ab664190628846cdec5aac118f8c627c50d24954496d2a67530a671045af488" exitCode=0
Jul 14 17:32:09.514984 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:09.514647    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67" event={"ID":"1dcf83b9-cc2d-466b-be3d-f6bfe1757234","Type":"ContainerDied","Data":"8ab664190628846cdec5aac118f8c627c50d24954496d2a67530a671045af488"}
Jul 14 17:32:10.519907 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:10.519823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67" event={"ID":"1dcf83b9-cc2d-466b-be3d-f6bfe1757234","Type":"ContainerStarted","Data":"14ef9713780b0aba83cabb3c525ef4b5959883964d6cdfd9e2310591c05913f1"}
Jul 14 17:32:10.520276 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:10.520183    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:10.542594 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:10.542544    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67" podStartSLOduration=3.109347379 podStartE2EDuration="7.542528527s" totalImagesPullingTime="3.359220893s" totalInitContainerRuntime="1.073960255s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:10.540671032 +0000 UTC m=+377.085074852" watchObservedRunningTime="2026-07-14 17:32:10.542528527 +0000 UTC m=+377.086932322"
Jul 14 17:32:16.503608 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:16.503581    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cb94d9664-qbq9j"
Jul 14 17:32:21.525400 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:21.525368    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f54d68b84-glz67"
Jul 14 17:32:23.875054 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.875022    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"]
Jul 14 17:32:23.971062 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.971023    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"]
Jul 14 17:32:23.971223 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.971125    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:23.973912 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.973881    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 17:32:23.973912 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.973879    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 17:32:23.974103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.973933    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 17:32:23.974103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.973970    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 17:32:23.974909 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:23.974894    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7vzf8\""
Jul 14 17:32:24.120388 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.115784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/745389ea-c983-4104-b107-09fcf138d169-tmp\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.120388 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.115922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zwd\" (UniqueName: \"kubernetes.io/projected/745389ea-c983-4104-b107-09fcf138d169-kube-api-access-w2zwd\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.120388 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.115979    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745389ea-c983-4104-b107-09fcf138d169-tls-certs\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.217151 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.217063    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745389ea-c983-4104-b107-09fcf138d169-tls-certs\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.217314 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.217192    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/745389ea-c983-4104-b107-09fcf138d169-tmp\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.217314 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.217258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zwd\" (UniqueName: \"kubernetes.io/projected/745389ea-c983-4104-b107-09fcf138d169-kube-api-access-w2zwd\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.219387 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.219361    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/745389ea-c983-4104-b107-09fcf138d169-tmp\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.219697 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.219675    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745389ea-c983-4104-b107-09fcf138d169-tls-certs\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.234142 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.234113    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zwd\" (UniqueName: \"kubernetes.io/projected/745389ea-c983-4104-b107-09fcf138d169-kube-api-access-w2zwd\") pod \"kube-auth-proxy-6dfddfddb9-s6qvh\" (UID: \"745389ea-c983-4104-b107-09fcf138d169\") " pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.281272 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.281245    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"
Jul 14 17:32:24.433028 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.432993    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh"]
Jul 14 17:32:24.436203 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:32:24.436171    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745389ea_c983_4104_b107_09fcf138d169.slice/crio-1f1992db6942aa662efcc623fe8b42af98885c5d79e33b0ac15afa37ce6f4823 WatchSource:0}: Error finding container 1f1992db6942aa662efcc623fe8b42af98885c5d79e33b0ac15afa37ce6f4823: Status 404 returned error can't find the container with id 1f1992db6942aa662efcc623fe8b42af98885c5d79e33b0ac15afa37ce6f4823
Jul 14 17:32:24.574590 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:24.574558    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh" event={"ID":"745389ea-c983-4104-b107-09fcf138d169","Type":"ContainerStarted","Data":"1f1992db6942aa662efcc623fe8b42af98885c5d79e33b0ac15afa37ce6f4823"}
Jul 14 17:32:27.589310 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:27.589276    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh" event={"ID":"745389ea-c983-4104-b107-09fcf138d169","Type":"ContainerStarted","Data":"07d0b33fdb84502885e0b35be11396d02cf9b689c9390f46423cb52fbdf39d98"}
Jul 14 17:32:27.605916 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:27.605873    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dfddfddb9-s6qvh" podStartSLOduration=1.837642487 podStartE2EDuration="4.605851877s" totalImagesPullingTime="2.76820939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:27.605170846 +0000 UTC m=+394.149574641" watchObservedRunningTime="2026-07-14 17:32:27.605851877 +0000 UTC m=+394.150255672"
Jul 14 17:32:33.663473 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.663437    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"]
Jul 14 17:32:33.665578 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.665557    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.668261 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.668242    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-95f6x\""
Jul 14 17:32:33.668594 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.668577    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 14 17:32:33.687667 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.687642    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"]
Jul 14 17:32:33.698550 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698523    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698668 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698600    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698668 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698644    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698776 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698670    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698776 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcab530e-7cac-465f-93dc-4ea18db00fca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698776 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698733    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698786    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698815    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlbv\" (UniqueName: \"kubernetes.io/projected/fcab530e-7cac-465f-93dc-4ea18db00fca-kube-api-access-qmlbv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.698898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.698870    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799217 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799191    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799392 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799253    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799392 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799392 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799300    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799392 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799322    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcab530e-7cac-465f-93dc-4ea18db00fca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799392 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799372    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799639 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799639 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799441    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlbv\" (UniqueName: \"kubernetes.io/projected/fcab530e-7cac-465f-93dc-4ea18db00fca-kube-api-access-qmlbv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799639 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799939 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799791    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.799939 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.800090 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799971    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.800090 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.799991    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.800258 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.800235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcab530e-7cac-465f-93dc-4ea18db00fca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.801697 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.801679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.801863 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.801847    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.815215 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.815190    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcab530e-7cac-465f-93dc-4ea18db00fca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.817717 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.817697    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlbv\" (UniqueName: \"kubernetes.io/projected/fcab530e-7cac-465f-93dc-4ea18db00fca-kube-api-access-qmlbv\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678dd9xp\" (UID: \"fcab530e-7cac-465f-93dc-4ea18db00fca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:33.975735 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:33.975651    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:34.106985 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.106958    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"]
Jul 14 17:32:34.109385 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:32:34.109328    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcab530e_7cac_465f_93dc_4ea18db00fca.slice/crio-6ae1073f4d25aa5ddd586fbbe5b3dff71053689b7eaac3af40cf05cee593c0b8 WatchSource:0}: Error finding container 6ae1073f4d25aa5ddd586fbbe5b3dff71053689b7eaac3af40cf05cee593c0b8: Status 404 returned error can't find the container with id 6ae1073f4d25aa5ddd586fbbe5b3dff71053689b7eaac3af40cf05cee593c0b8
Jul 14 17:32:34.620052 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.620008    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp" event={"ID":"fcab530e-7cac-465f-93dc-4ea18db00fca","Type":"ContainerStarted","Data":"6ae1073f4d25aa5ddd586fbbe5b3dff71053689b7eaac3af40cf05cee593c0b8"}
Jul 14 17:32:34.647223 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.647192    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"]
Jul 14 17:32:34.650202 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.650181    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:34.652938 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.652913    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 14 17:32:34.653046 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.652960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-tlm47\""
Jul 14 17:32:34.662485 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.662460    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"]
Jul 14 17:32:34.705134 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.705103    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316a6d30-910a-441e-9d41-dd36d556ee68-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:34.705471 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.705155    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvxzj\" (UniqueName: \"kubernetes.io/projected/316a6d30-910a-441e-9d41-dd36d556ee68-kube-api-access-xvxzj\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:34.810806 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.810161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvxzj\" (UniqueName: \"kubernetes.io/projected/316a6d30-910a-441e-9d41-dd36d556ee68-kube-api-access-xvxzj\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:34.810806 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.810281    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316a6d30-910a-441e-9d41-dd36d556ee68-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:34.810806 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:32:34.810534    2567 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 14 17:32:34.810806 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:32:34.810618    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/316a6d30-910a-441e-9d41-dd36d556ee68-cert podName:316a6d30-910a-441e-9d41-dd36d556ee68 nodeName:}" failed. No retries permitted until 2026-07-14 17:32:35.310593872 +0000 UTC m=+401.854997647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/316a6d30-910a-441e-9d41-dd36d556ee68-cert") pod "llmisvc-controller-manager-777c6bd7d5-42b9r" (UID: "316a6d30-910a-441e-9d41-dd36d556ee68") : secret "llmisvc-webhook-server-cert" not found
Jul 14 17:32:34.830689 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:34.830572    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvxzj\" (UniqueName: \"kubernetes.io/projected/316a6d30-910a-441e-9d41-dd36d556ee68-kube-api-access-xvxzj\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:35.315253 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:35.314674    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316a6d30-910a-441e-9d41-dd36d556ee68-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:35.320313 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:35.318487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/316a6d30-910a-441e-9d41-dd36d556ee68-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-42b9r\" (UID: \"316a6d30-910a-441e-9d41-dd36d556ee68\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:35.560714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:35.560673    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:35.710978 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:35.710954    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"]
Jul 14 17:32:35.713974 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:32:35.713939    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod316a6d30_910a_441e_9d41_dd36d556ee68.slice/crio-3f5600d5e2a27dbce76bd120ecd233e0362e82fdaa5e5c0377255761748b32ee WatchSource:0}: Error finding container 3f5600d5e2a27dbce76bd120ecd233e0362e82fdaa5e5c0377255761748b32ee: Status 404 returned error can't find the container with id 3f5600d5e2a27dbce76bd120ecd233e0362e82fdaa5e5c0377255761748b32ee
Jul 14 17:32:36.632588 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:36.632543    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r" event={"ID":"316a6d30-910a-441e-9d41-dd36d556ee68","Type":"ContainerStarted","Data":"3f5600d5e2a27dbce76bd120ecd233e0362e82fdaa5e5c0377255761748b32ee"}
Jul 14 17:32:36.717629 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:36.717590    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:32:36.717864 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:36.717673    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:32:36.717864 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:36.717702    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:32:37.640228 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:37.640179    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp" event={"ID":"fcab530e-7cac-465f-93dc-4ea18db00fca","Type":"ContainerStarted","Data":"de411abd2b7851b0aebff9a43e2d92664654c327679cf46d84578c5796329fc1"}
Jul 14 17:32:37.661956 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:37.661902    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp" podStartSLOduration=2.055735493 podStartE2EDuration="4.661880758s" totalImagesPullingTime="2.606145265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:37.659902751 +0000 UTC m=+404.204306559" watchObservedRunningTime="2026-07-14 17:32:37.661880758 +0000 UTC m=+404.206284554"
Jul 14 17:32:37.976046 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:37.975944    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:37.980713 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:37.980692    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:38.645108 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:38.645077    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:38.646124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:38.646101    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678dd9xp"
Jul 14 17:32:39.650095 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:39.650062    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r" event={"ID":"316a6d30-910a-441e-9d41-dd36d556ee68","Type":"ContainerStarted","Data":"07335dfb7e02bf31469d3db2da50841d28dce6e192290954088b623c3f2505bb"}
Jul 14 17:32:39.650511 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:39.650124    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:32:39.669367 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:32:39.669298    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r" podStartSLOduration=1.961310012 podStartE2EDuration="5.669281366s" totalImagesPullingTime="3.707971354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:32:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:32:39.667008607 +0000 UTC m=+406.211412427" watchObservedRunningTime="2026-07-14 17:32:39.669281366 +0000 UTC m=+406.213685186"
Jul 14 17:33:10.656524 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:33:10.656492    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-42b9r"
Jul 14 17:34:24.693778 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.693742    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"]
Jul 14 17:34:24.697108 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.697087    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:24.700002 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.699984    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 17:34:24.700122 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.699996    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kkk56\""
Jul 14 17:34:24.700173 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.700149    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 17:34:24.710053 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.710032    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"]
Jul 14 17:34:24.737880 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.737859    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:24.737983 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.737895    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltxm\" (UniqueName: \"kubernetes.io/projected/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-kube-api-access-dltxm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:24.839268 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.839230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:24.839268 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.839277    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltxm\" (UniqueName: \"kubernetes.io/projected/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-kube-api-access-dltxm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:24.839631 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.839612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:24.848022 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:24.847997    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltxm\" (UniqueName: \"kubernetes.io/projected/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-kube-api-access-dltxm\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:25.007223 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.007136    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:25.134710 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.134686    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"]
Jul 14 17:34:25.137996 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:34:25.137969    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dc63fe_b99e_44cc_b4df_0a673aff75e3.slice/crio-54640bb8ec6ec396bc11867b04f0f11e6cc94e5795bf141c80aa13750c08ff0c WatchSource:0}: Error finding container 54640bb8ec6ec396bc11867b04f0f11e6cc94e5795bf141c80aa13750c08ff0c: Status 404 returned error can't find the container with id 54640bb8ec6ec396bc11867b04f0f11e6cc94e5795bf141c80aa13750c08ff0c
Jul 14 17:34:25.440584 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.440553    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"]
Jul 14 17:34:25.444166 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.444140    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"]
Jul 14 17:34:25.518308 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.518273    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"]
Jul 14 17:34:25.522733 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.522707    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:34:25.526177 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.526155    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tbbxc\""
Jul 14 17:34:25.533791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.533768    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"]
Jul 14 17:34:25.546385 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.546361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8dd\" (UniqueName: \"kubernetes.io/projected/2521423d-6155-4be1-b344-35d5cbecbd62-kube-api-access-2t8dd\") pod \"limitador-operator-controller-manager-859d7b78bd-dhmrh\" (UID: \"2521423d-6155-4be1-b344-35d5cbecbd62\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:34:25.647733 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.647703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8dd\" (UniqueName: \"kubernetes.io/projected/2521423d-6155-4be1-b344-35d5cbecbd62-kube-api-access-2t8dd\") pod \"limitador-operator-controller-manager-859d7b78bd-dhmrh\" (UID: \"2521423d-6155-4be1-b344-35d5cbecbd62\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:34:25.656765 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.656734    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8dd\" (UniqueName: \"kubernetes.io/projected/2521423d-6155-4be1-b344-35d5cbecbd62-kube-api-access-2t8dd\") pod \"limitador-operator-controller-manager-859d7b78bd-dhmrh\" (UID: \"2521423d-6155-4be1-b344-35d5cbecbd62\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:34:25.834271 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.834240    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:34:25.967620 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:25.967588    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"]
Jul 14 17:34:25.971786 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:34:25.971756    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2521423d_6155_4be1_b344_35d5cbecbd62.slice/crio-bc7b2732f801513ccc132116b5ccf39b3ece960beb6c06b47054cab81e48dc22 WatchSource:0}: Error finding container bc7b2732f801513ccc132116b5ccf39b3ece960beb6c06b47054cab81e48dc22: Status 404 returned error can't find the container with id bc7b2732f801513ccc132116b5ccf39b3ece960beb6c06b47054cab81e48dc22
Jul 14 17:34:26.092177 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:26.092087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh" event={"ID":"2521423d-6155-4be1-b344-35d5cbecbd62","Type":"ContainerStarted","Data":"bc7b2732f801513ccc132116b5ccf39b3ece960beb6c06b47054cab81e48dc22"}
Jul 14 17:34:29.107617 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:29.107576    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh" event={"ID":"2521423d-6155-4be1-b344-35d5cbecbd62","Type":"ContainerStarted","Data":"2da0648e742d1c5520e34500d10bbb160239562d1e912fdec3647153e5504e27"}
Jul 14 17:34:29.108105 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:29.107649    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:34:29.125276 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:29.125229    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh" podStartSLOduration=1.476479126 podStartE2EDuration="4.125209553s" totalImagesPullingTime="2.648730427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:34:29.123214495 +0000 UTC m=+515.667618302" watchObservedRunningTime="2026-07-14 17:34:29.125209553 +0000 UTC m=+515.669613347"
Jul 14 17:34:31.119371 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.119321    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" containerName="manager" containerID="cri-o://3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029" gracePeriod=2
Jul 14 17:34:31.354857 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.354830    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:31.357272 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:34:31.357247    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" is forbidden: User \"system:node:ip-10-0-138-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-98.ec2.internal' and this object" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:31.400196 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.400168    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dltxm\" (UniqueName: \"kubernetes.io/projected/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-kube-api-access-dltxm\") pod \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") "
Jul 14 17:34:31.400316 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.400239    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-extensions-socket-volume\") pod \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\" (UID: \"a5dc63fe-b99e-44cc-b4df-0a673aff75e3\") "
Jul 14 17:34:31.400490 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.400466    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a5dc63fe-b99e-44cc-b4df-0a673aff75e3" (UID: "a5dc63fe-b99e-44cc-b4df-0a673aff75e3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:34:31.400539 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.400532    2567 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-extensions-socket-volume\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:34:31.402202 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.402179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-kube-api-access-dltxm" (OuterVolumeSpecName: "kube-api-access-dltxm") pod "a5dc63fe-b99e-44cc-b4df-0a673aff75e3" (UID: "a5dc63fe-b99e-44cc-b4df-0a673aff75e3"). InnerVolumeSpecName "kube-api-access-dltxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:34:31.501536 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:31.501480    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltxm\" (UniqueName: \"kubernetes.io/projected/a5dc63fe-b99e-44cc-b4df-0a673aff75e3-kube-api-access-dltxm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:34:32.051404 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:32.051368    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" path="/var/lib/kubelet/pods/a5dc63fe-b99e-44cc-b4df-0a673aff75e3/volumes"
Jul 14 17:34:32.124430 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:32.124392    2567 generic.go:356] "Generic (PLEG): container finished" podID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" containerID="3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029" exitCode=2
Jul 14 17:34:32.124849 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:32.124440    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:32.124849 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:32.124443    2567 scope.go:117] "RemoveContainer" containerID="3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029"
Jul 14 17:34:32.129132 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:34:32.129103    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" is forbidden: User \"system:node:ip-10-0-138-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-98.ec2.internal' and this object" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:32.133065 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:32.133041    2567 scope.go:117] "RemoveContainer" containerID="3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029"
Jul 14 17:34:32.133360 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:34:32.133322    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029\": container with ID starting with 3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029 not found: ID does not exist" containerID="3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029"
Jul 14 17:34:32.133422 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:32.133371    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029"} err="failed to get container status \"3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029\": rpc error: code = NotFound desc = could not find container \"3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029\": container with ID starting with 3cbd302c1565a6358b23b994c84bf68104097c85cd232922b63c324977c29029 not found: ID does not exist"
Jul 14 17:34:34.050839 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:34:34.050805    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-xbmx9\" is forbidden: User \"system:node:ip-10-0-138-98.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-98.ec2.internal' and this object" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xbmx9"
Jul 14 17:34:40.115030 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:34:40.115004    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-dhmrh"
Jul 14 17:35:20.200869 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.200836    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8kjq6"]
Jul 14 17:35:20.201327 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.201153    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" containerName="manager"
Jul 14 17:35:20.201327 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.201164    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" containerName="manager"
Jul 14 17:35:20.201327 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.201217    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5dc63fe-b99e-44cc-b4df-0a673aff75e3" containerName="manager"
Jul 14 17:35:20.203700 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.203684    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:20.206471 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.206439    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sps47\""
Jul 14 17:35:20.211871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.211850    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8kjq6"]
Jul 14 17:35:20.319540 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.319508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6znp\" (UniqueName: \"kubernetes.io/projected/502d76ba-b9ce-452e-83b1-64142632bff8-kube-api-access-j6znp\") pod \"authorino-58cd955675-8kjq6\" (UID: \"502d76ba-b9ce-452e-83b1-64142632bff8\") " pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:20.420720 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.420694    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6znp\" (UniqueName: \"kubernetes.io/projected/502d76ba-b9ce-452e-83b1-64142632bff8-kube-api-access-j6znp\") pod \"authorino-58cd955675-8kjq6\" (UID: \"502d76ba-b9ce-452e-83b1-64142632bff8\") " pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:20.428772 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.428746    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6znp\" (UniqueName: \"kubernetes.io/projected/502d76ba-b9ce-452e-83b1-64142632bff8-kube-api-access-j6znp\") pod \"authorino-58cd955675-8kjq6\" (UID: \"502d76ba-b9ce-452e-83b1-64142632bff8\") " pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:20.523819 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.523753    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:20.565862 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.565827    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-z74cr"]
Jul 14 17:35:20.570240 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.570218    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:20.576156 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.576096    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-z74cr"]
Jul 14 17:35:20.622904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.622867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmf2\" (UniqueName: \"kubernetes.io/projected/a7217858-8570-4f5b-8df9-fc6ea3d3993d-kube-api-access-7cmf2\") pod \"authorino-75d556bd77-z74cr\" (UID: \"a7217858-8570-4f5b-8df9-fc6ea3d3993d\") " pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:20.650830 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.650792    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8kjq6"]
Jul 14 17:35:20.653040 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:35:20.653013    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod502d76ba_b9ce_452e_83b1_64142632bff8.slice/crio-0f149593f469920be190945e5bada6b274b2926eaa2c5c2f92647b4f6d447697 WatchSource:0}: Error finding container 0f149593f469920be190945e5bada6b274b2926eaa2c5c2f92647b4f6d447697: Status 404 returned error can't find the container with id 0f149593f469920be190945e5bada6b274b2926eaa2c5c2f92647b4f6d447697
Jul 14 17:35:20.724078 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.724043    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmf2\" (UniqueName: \"kubernetes.io/projected/a7217858-8570-4f5b-8df9-fc6ea3d3993d-kube-api-access-7cmf2\") pod \"authorino-75d556bd77-z74cr\" (UID: \"a7217858-8570-4f5b-8df9-fc6ea3d3993d\") " pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:20.732680 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.732657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmf2\" (UniqueName: \"kubernetes.io/projected/a7217858-8570-4f5b-8df9-fc6ea3d3993d-kube-api-access-7cmf2\") pod \"authorino-75d556bd77-z74cr\" (UID: \"a7217858-8570-4f5b-8df9-fc6ea3d3993d\") " pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:20.882124 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:20.882090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:21.001161 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:21.001138    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-z74cr"]
Jul 14 17:35:21.003209 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:35:21.003182    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7217858_8570_4f5b_8df9_fc6ea3d3993d.slice/crio-64175125e6886935f79fb58750419599ce5aadb9f6f05e1179f899522002ee3e WatchSource:0}: Error finding container 64175125e6886935f79fb58750419599ce5aadb9f6f05e1179f899522002ee3e: Status 404 returned error can't find the container with id 64175125e6886935f79fb58750419599ce5aadb9f6f05e1179f899522002ee3e
Jul 14 17:35:21.332264 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:21.332167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-z74cr" event={"ID":"a7217858-8570-4f5b-8df9-fc6ea3d3993d","Type":"ContainerStarted","Data":"64175125e6886935f79fb58750419599ce5aadb9f6f05e1179f899522002ee3e"}
Jul 14 17:35:21.334491 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:21.334365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8kjq6" event={"ID":"502d76ba-b9ce-452e-83b1-64142632bff8","Type":"ContainerStarted","Data":"0f149593f469920be190945e5bada6b274b2926eaa2c5c2f92647b4f6d447697"}
Jul 14 17:35:24.354511 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:24.354466    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8kjq6" event={"ID":"502d76ba-b9ce-452e-83b1-64142632bff8","Type":"ContainerStarted","Data":"bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0"}
Jul 14 17:35:24.356774 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:24.356741    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-z74cr" event={"ID":"a7217858-8570-4f5b-8df9-fc6ea3d3993d","Type":"ContainerStarted","Data":"835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae"}
Jul 14 17:35:24.370028 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:24.369985    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8kjq6" podStartSLOduration=0.80913699 podStartE2EDuration="4.369970774s" totalImagesPullingTime="3.560833784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:35:24.368926586 +0000 UTC m=+570.913330382" watchObservedRunningTime="2026-07-14 17:35:24.369970774 +0000 UTC m=+570.914374567"
Jul 14 17:35:24.384296 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:24.384240    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-z74cr" podStartSLOduration=1.186252916 podStartE2EDuration="4.384227427s" totalImagesPullingTime="3.197974511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:35:24.382715076 +0000 UTC m=+570.927118871" watchObservedRunningTime="2026-07-14 17:35:24.384227427 +0000 UTC m=+570.928631220"
Jul 14 17:35:24.405985 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:24.405954    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8kjq6"]
Jul 14 17:35:26.364577 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:26.364535    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8kjq6" podUID="502d76ba-b9ce-452e-83b1-64142632bff8" containerName="authorino" containerID="cri-o://bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0" gracePeriod=30
Jul 14 17:35:26.602904 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:26.602881    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:26.677264 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:26.677190    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6znp\" (UniqueName: \"kubernetes.io/projected/502d76ba-b9ce-452e-83b1-64142632bff8-kube-api-access-j6znp\") pod \"502d76ba-b9ce-452e-83b1-64142632bff8\" (UID: \"502d76ba-b9ce-452e-83b1-64142632bff8\") "
Jul 14 17:35:26.679202 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:26.679172    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502d76ba-b9ce-452e-83b1-64142632bff8-kube-api-access-j6znp" (OuterVolumeSpecName: "kube-api-access-j6znp") pod "502d76ba-b9ce-452e-83b1-64142632bff8" (UID: "502d76ba-b9ce-452e-83b1-64142632bff8"). InnerVolumeSpecName "kube-api-access-j6znp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:35:26.778037 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:26.778003    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6znp\" (UniqueName: \"kubernetes.io/projected/502d76ba-b9ce-452e-83b1-64142632bff8-kube-api-access-j6znp\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:35:27.370019 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.369979    2567 generic.go:356] "Generic (PLEG): container finished" podID="502d76ba-b9ce-452e-83b1-64142632bff8" containerID="bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0" exitCode=0
Jul 14 17:35:27.370019 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.370028    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8kjq6"
Jul 14 17:35:27.370621 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.370066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8kjq6" event={"ID":"502d76ba-b9ce-452e-83b1-64142632bff8","Type":"ContainerDied","Data":"bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0"}
Jul 14 17:35:27.370621 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.370113    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8kjq6" event={"ID":"502d76ba-b9ce-452e-83b1-64142632bff8","Type":"ContainerDied","Data":"0f149593f469920be190945e5bada6b274b2926eaa2c5c2f92647b4f6d447697"}
Jul 14 17:35:27.370621 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.370140    2567 scope.go:117] "RemoveContainer" containerID="bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0"
Jul 14 17:35:27.387666 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.387639    2567 scope.go:117] "RemoveContainer" containerID="bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0"
Jul 14 17:35:27.388095 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:35:27.388047    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0\": container with ID starting with bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0 not found: ID does not exist" containerID="bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0"
Jul 14 17:35:27.388176 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.388090    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0"} err="failed to get container status \"bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0\": rpc error: code = NotFound desc = could not find container \"bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0\": container with ID starting with bacee92a0389723b03688a14e00ec2358e312485873e34fd4b8079f630d3acf0 not found: ID does not exist"
Jul 14 17:35:27.397391 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.397364    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8kjq6"]
Jul 14 17:35:27.403383 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:27.403360    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8kjq6"]
Jul 14 17:35:28.051401 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:28.051365    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="502d76ba-b9ce-452e-83b1-64142632bff8" path="/var/lib/kubelet/pods/502d76ba-b9ce-452e-83b1-64142632bff8/volumes"
Jul 14 17:35:48.739667 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.739634    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwrft"]
Jul 14 17:35:48.740032 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.739953    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="502d76ba-b9ce-452e-83b1-64142632bff8" containerName="authorino"
Jul 14 17:35:48.740032 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.739965    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="502d76ba-b9ce-452e-83b1-64142632bff8" containerName="authorino"
Jul 14 17:35:48.740032 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.740019    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="502d76ba-b9ce-452e-83b1-64142632bff8" containerName="authorino"
Jul 14 17:35:48.742865 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.742849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:48.746997 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.746969    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwrft"]
Jul 14 17:35:48.878916 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.878867    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jjr\" (UniqueName: \"kubernetes.io/projected/6f718250-5fe4-4e0b-94c5-79198d075177-kube-api-access-t9jjr\") pod \"authorino-5d4b67c446-wwrft\" (UID: \"6f718250-5fe4-4e0b-94c5-79198d075177\") " pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:48.980022 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.979981    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jjr\" (UniqueName: \"kubernetes.io/projected/6f718250-5fe4-4e0b-94c5-79198d075177-kube-api-access-t9jjr\") pod \"authorino-5d4b67c446-wwrft\" (UID: \"6f718250-5fe4-4e0b-94c5-79198d075177\") " pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:48.989128 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.989098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jjr\" (UniqueName: \"kubernetes.io/projected/6f718250-5fe4-4e0b-94c5-79198d075177-kube-api-access-t9jjr\") pod \"authorino-5d4b67c446-wwrft\" (UID: \"6f718250-5fe4-4e0b-94c5-79198d075177\") " pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:48.995644 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.995573    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwrft"]
Jul 14 17:35:48.995848 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:48.995836    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:49.023948 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.023911    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-544587b7c5-2lvgt"]
Jul 14 17:35:49.028890 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.028862    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:49.033811 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.033784    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-544587b7c5-2lvgt"]
Jul 14 17:35:49.130021 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.129990    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwrft"]
Jul 14 17:35:49.134058 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:35:49.134030    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f718250_5fe4_4e0b_94c5_79198d075177.slice/crio-6348dfb4b207f020520ec7acf4c42414549c12022f7105d7ec5113aa43caf950 WatchSource:0}: Error finding container 6348dfb4b207f020520ec7acf4c42414549c12022f7105d7ec5113aa43caf950: Status 404 returned error can't find the container with id 6348dfb4b207f020520ec7acf4c42414549c12022f7105d7ec5113aa43caf950
Jul 14 17:35:49.182332 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.182300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869jc\" (UniqueName: \"kubernetes.io/projected/6258a2e9-e5f7-42ec-8802-533601dac0c7-kube-api-access-869jc\") pod \"authorino-544587b7c5-2lvgt\" (UID: \"6258a2e9-e5f7-42ec-8802-533601dac0c7\") " pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:49.283787 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.283700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869jc\" (UniqueName: \"kubernetes.io/projected/6258a2e9-e5f7-42ec-8802-533601dac0c7-kube-api-access-869jc\") pod \"authorino-544587b7c5-2lvgt\" (UID: \"6258a2e9-e5f7-42ec-8802-533601dac0c7\") " pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:49.291850 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.291823    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544587b7c5-2lvgt"]
Jul 14 17:35:49.292084 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:35:49.292062    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-869jc], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-544587b7c5-2lvgt" podUID="6258a2e9-e5f7-42ec-8802-533601dac0c7"
Jul 14 17:35:49.295057 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.295040    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-869jc\" (UniqueName: \"kubernetes.io/projected/6258a2e9-e5f7-42ec-8802-533601dac0c7-kube-api-access-869jc\") pod \"authorino-544587b7c5-2lvgt\" (UID: \"6258a2e9-e5f7-42ec-8802-533601dac0c7\") " pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:49.319335 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.319310    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59d487b9c5-gls2f"]
Jul 14 17:35:49.322678 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.322663    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.325288 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.325272    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 14 17:35:49.331455 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.331431    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59d487b9c5-gls2f"]
Jul 14 17:35:49.469435 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.469384    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wwrft" event={"ID":"6f718250-5fe4-4e0b-94c5-79198d075177","Type":"ContainerStarted","Data":"6348dfb4b207f020520ec7acf4c42414549c12022f7105d7ec5113aa43caf950"}
Jul 14 17:35:49.469435 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.469419    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:49.475656 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.475633    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:49.485717 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.485695    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwlc\" (UniqueName: \"kubernetes.io/projected/b749ffae-edad-4915-9dc8-ec15c63b596b-kube-api-access-bwwlc\") pod \"authorino-59d487b9c5-gls2f\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") " pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.485805 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.485740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b749ffae-edad-4915-9dc8-ec15c63b596b-tls-cert\") pod \"authorino-59d487b9c5-gls2f\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") " pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.586335 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.586307    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-869jc\" (UniqueName: \"kubernetes.io/projected/6258a2e9-e5f7-42ec-8802-533601dac0c7-kube-api-access-869jc\") pod \"6258a2e9-e5f7-42ec-8802-533601dac0c7\" (UID: \"6258a2e9-e5f7-42ec-8802-533601dac0c7\") "
Jul 14 17:35:49.586554 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.586534    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b749ffae-edad-4915-9dc8-ec15c63b596b-tls-cert\") pod \"authorino-59d487b9c5-gls2f\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") " pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.586687 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.586670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwlc\" (UniqueName: \"kubernetes.io/projected/b749ffae-edad-4915-9dc8-ec15c63b596b-kube-api-access-bwwlc\") pod \"authorino-59d487b9c5-gls2f\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") " pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.588410 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.588387    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6258a2e9-e5f7-42ec-8802-533601dac0c7-kube-api-access-869jc" (OuterVolumeSpecName: "kube-api-access-869jc") pod "6258a2e9-e5f7-42ec-8802-533601dac0c7" (UID: "6258a2e9-e5f7-42ec-8802-533601dac0c7"). InnerVolumeSpecName "kube-api-access-869jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:35:49.588815 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.588799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b749ffae-edad-4915-9dc8-ec15c63b596b-tls-cert\") pod \"authorino-59d487b9c5-gls2f\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") " pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.596758 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.596735    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwlc\" (UniqueName: \"kubernetes.io/projected/b749ffae-edad-4915-9dc8-ec15c63b596b-kube-api-access-bwwlc\") pod \"authorino-59d487b9c5-gls2f\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") " pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.631878 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.631857    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:35:49.688064 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.687939    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869jc\" (UniqueName: \"kubernetes.io/projected/6258a2e9-e5f7-42ec-8802-533601dac0c7-kube-api-access-869jc\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:35:49.765895 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:49.765857    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59d487b9c5-gls2f"]
Jul 14 17:35:49.768218 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:35:49.768184    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb749ffae_edad_4915_9dc8_ec15c63b596b.slice/crio-003e89f6500fe1e21ddbfca36907204d692307d808a94c9eb4013590d21f3293 WatchSource:0}: Error finding container 003e89f6500fe1e21ddbfca36907204d692307d808a94c9eb4013590d21f3293: Status 404 returned error can't find the container with id 003e89f6500fe1e21ddbfca36907204d692307d808a94c9eb4013590d21f3293
Jul 14 17:35:50.476187 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.476145    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wwrft" event={"ID":"6f718250-5fe4-4e0b-94c5-79198d075177","Type":"ContainerStarted","Data":"474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3"}
Jul 14 17:35:50.476419 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.476220    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-wwrft" podUID="6f718250-5fe4-4e0b-94c5-79198d075177" containerName="authorino" containerID="cri-o://474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3" gracePeriod=30
Jul 14 17:35:50.477788 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.477737    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59d487b9c5-gls2f" event={"ID":"b749ffae-edad-4915-9dc8-ec15c63b596b","Type":"ContainerStarted","Data":"9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89"}
Jul 14 17:35:50.477788 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.477767    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-544587b7c5-2lvgt"
Jul 14 17:35:50.477788 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.477777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59d487b9c5-gls2f" event={"ID":"b749ffae-edad-4915-9dc8-ec15c63b596b","Type":"ContainerStarted","Data":"003e89f6500fe1e21ddbfca36907204d692307d808a94c9eb4013590d21f3293"}
Jul 14 17:35:50.505689 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.505645    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59d487b9c5-gls2f" podStartSLOduration=1.106881893 podStartE2EDuration="1.50563126s" totalImagesPullingTime="398.749367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:35:50.504548805 +0000 UTC m=+597.048952612" watchObservedRunningTime="2026-07-14 17:35:50.50563126 +0000 UTC m=+597.050035055"
Jul 14 17:35:50.505891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.505872    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-wwrft" podStartSLOduration=2.05461134 podStartE2EDuration="2.50586534s" totalImagesPullingTime="451.254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:35:50.490703726 +0000 UTC m=+597.035107523" watchObservedRunningTime="2026-07-14 17:35:50.50586534 +0000 UTC m=+597.050269136"
Jul 14 17:35:50.529126 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.529097    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-544587b7c5-2lvgt"]
Jul 14 17:35:50.534501 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.534477    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-z74cr"]
Jul 14 17:35:50.534691 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.534661    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-z74cr" podUID="a7217858-8570-4f5b-8df9-fc6ea3d3993d" containerName="authorino" containerID="cri-o://835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae" gracePeriod=30
Jul 14 17:35:50.536160 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.536137    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-544587b7c5-2lvgt"]
Jul 14 17:35:50.778104 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.778079    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:50.785821 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.785798    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:50.795151 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.795128    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9jjr\" (UniqueName: \"kubernetes.io/projected/6f718250-5fe4-4e0b-94c5-79198d075177-kube-api-access-t9jjr\") pod \"6f718250-5fe4-4e0b-94c5-79198d075177\" (UID: \"6f718250-5fe4-4e0b-94c5-79198d075177\") "
Jul 14 17:35:50.795327 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.795158    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cmf2\" (UniqueName: \"kubernetes.io/projected/a7217858-8570-4f5b-8df9-fc6ea3d3993d-kube-api-access-7cmf2\") pod \"a7217858-8570-4f5b-8df9-fc6ea3d3993d\" (UID: \"a7217858-8570-4f5b-8df9-fc6ea3d3993d\") "
Jul 14 17:35:50.797648 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.797614    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f718250-5fe4-4e0b-94c5-79198d075177-kube-api-access-t9jjr" (OuterVolumeSpecName: "kube-api-access-t9jjr") pod "6f718250-5fe4-4e0b-94c5-79198d075177" (UID: "6f718250-5fe4-4e0b-94c5-79198d075177"). InnerVolumeSpecName "kube-api-access-t9jjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:35:50.797739 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.797676    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7217858-8570-4f5b-8df9-fc6ea3d3993d-kube-api-access-7cmf2" (OuterVolumeSpecName: "kube-api-access-7cmf2") pod "a7217858-8570-4f5b-8df9-fc6ea3d3993d" (UID: "a7217858-8570-4f5b-8df9-fc6ea3d3993d"). InnerVolumeSpecName "kube-api-access-7cmf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:35:50.895714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.895682    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9jjr\" (UniqueName: \"kubernetes.io/projected/6f718250-5fe4-4e0b-94c5-79198d075177-kube-api-access-t9jjr\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:35:50.895714 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:50.895706    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cmf2\" (UniqueName: \"kubernetes.io/projected/a7217858-8570-4f5b-8df9-fc6ea3d3993d-kube-api-access-7cmf2\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:35:51.484535 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.484497    2567 generic.go:356] "Generic (PLEG): container finished" podID="6f718250-5fe4-4e0b-94c5-79198d075177" containerID="474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3" exitCode=0
Jul 14 17:35:51.484535 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.484536    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wwrft"
Jul 14 17:35:51.484847 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.484583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wwrft" event={"ID":"6f718250-5fe4-4e0b-94c5-79198d075177","Type":"ContainerDied","Data":"474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3"}
Jul 14 17:35:51.484847 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.484626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wwrft" event={"ID":"6f718250-5fe4-4e0b-94c5-79198d075177","Type":"ContainerDied","Data":"6348dfb4b207f020520ec7acf4c42414549c12022f7105d7ec5113aa43caf950"}
Jul 14 17:35:51.484847 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.484647    2567 scope.go:117] "RemoveContainer" containerID="474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3"
Jul 14 17:35:51.486016 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.485887    2567 generic.go:356] "Generic (PLEG): container finished" podID="a7217858-8570-4f5b-8df9-fc6ea3d3993d" containerID="835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae" exitCode=0
Jul 14 17:35:51.486016 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.485926    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-z74cr"
Jul 14 17:35:51.486016 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.485977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-z74cr" event={"ID":"a7217858-8570-4f5b-8df9-fc6ea3d3993d","Type":"ContainerDied","Data":"835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae"}
Jul 14 17:35:51.486016 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.486014    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-z74cr" event={"ID":"a7217858-8570-4f5b-8df9-fc6ea3d3993d","Type":"ContainerDied","Data":"64175125e6886935f79fb58750419599ce5aadb9f6f05e1179f899522002ee3e"}
Jul 14 17:35:51.493114 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.493091    2567 scope.go:117] "RemoveContainer" containerID="474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3"
Jul 14 17:35:51.493636 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:35:51.493357    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3\": container with ID starting with 474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3 not found: ID does not exist" containerID="474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3"
Jul 14 17:35:51.493636 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.493387    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3"} err="failed to get container status \"474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3\": rpc error: code = NotFound desc = could not find container \"474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3\": container with ID starting with 474ec2f8b058a3f51e32534884954d48d987b2b1dc2cb90ef97aad3ca21625d3 not found: ID does not exist"
Jul 14 17:35:51.493636 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.493406    2567 scope.go:117] "RemoveContainer" containerID="835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae"
Jul 14 17:35:51.500727 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.500712    2567 scope.go:117] "RemoveContainer" containerID="835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae"
Jul 14 17:35:51.500965 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:35:51.500948    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae\": container with ID starting with 835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae not found: ID does not exist" containerID="835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae"
Jul 14 17:35:51.501023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.500970    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae"} err="failed to get container status \"835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae\": rpc error: code = NotFound desc = could not find container \"835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae\": container with ID starting with 835d423026b3d0526412ac1ae6f23924b15bc8a96ad0b1e8a8f80f18f758f0ae not found: ID does not exist"
Jul 14 17:35:51.507304 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.507281    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-z74cr"]
Jul 14 17:35:51.511334 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.511315    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-z74cr"]
Jul 14 17:35:51.521095 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.521074    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwrft"]
Jul 14 17:35:51.524683 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:51.524661    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wwrft"]
Jul 14 17:35:52.052382 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:52.052334    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6258a2e9-e5f7-42ec-8802-533601dac0c7" path="/var/lib/kubelet/pods/6258a2e9-e5f7-42ec-8802-533601dac0c7/volumes"
Jul 14 17:35:52.052728 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:52.052559    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f718250-5fe4-4e0b-94c5-79198d075177" path="/var/lib/kubelet/pods/6f718250-5fe4-4e0b-94c5-79198d075177/volumes"
Jul 14 17:35:52.052831 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:52.052818    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7217858-8570-4f5b-8df9-fc6ea3d3993d" path="/var/lib/kubelet/pods/a7217858-8570-4f5b-8df9-fc6ea3d3993d/volumes"
Jul 14 17:35:53.827307 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:53.827280    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:35:53.828324 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:35:53.828299    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:38:05.176393 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176299    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"]
Jul 14 17:38:05.176853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176660    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f718250-5fe4-4e0b-94c5-79198d075177" containerName="authorino"
Jul 14 17:38:05.176853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176673    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f718250-5fe4-4e0b-94c5-79198d075177" containerName="authorino"
Jul 14 17:38:05.176853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176688    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7217858-8570-4f5b-8df9-fc6ea3d3993d" containerName="authorino"
Jul 14 17:38:05.176853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176697    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7217858-8570-4f5b-8df9-fc6ea3d3993d" containerName="authorino"
Jul 14 17:38:05.176853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176773    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7217858-8570-4f5b-8df9-fc6ea3d3993d" containerName="authorino"
Jul 14 17:38:05.176853 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.176782    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f718250-5fe4-4e0b-94c5-79198d075177" containerName="authorino"
Jul 14 17:38:05.179785 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.179768    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.182421 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.182399    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 17:38:05.182519 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.182423    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 17:38:05.183601 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.183581    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w8snq\""
Jul 14 17:38:05.183708 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.183682    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 14 17:38:05.188948 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.188924    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"]
Jul 14 17:38:05.273445 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.273413    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.273599 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.273449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.273599 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.273481    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.273599 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.273528    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.273599 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.273560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlfh\" (UniqueName: \"kubernetes.io/projected/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-kube-api-access-4tlfh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.273752 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.273619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374545 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374571    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374589    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374619    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlfh\" (UniqueName: \"kubernetes.io/projected/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-kube-api-access-4tlfh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.374979 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374960    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.375036 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.374989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.375076 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.375055    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.376936 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.376915    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.377023 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.376957    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.383143 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.383120    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlfh\" (UniqueName: \"kubernetes.io/projected/dae2cca9-3da7-426f-887f-1e4c3e6cda8c-kube-api-access-4tlfh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-77czt\" (UID: \"dae2cca9-3da7-426f-887f-1e4c3e6cda8c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.491199 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.491129    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:05.617334 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.617307    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"]
Jul 14 17:38:05.619753 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:38:05.619729    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae2cca9_3da7_426f_887f_1e4c3e6cda8c.slice/crio-cc04cdf178e15542b5e4238f4c52d967609a92437938d7cbf22dd79a0192c37e WatchSource:0}: Error finding container cc04cdf178e15542b5e4238f4c52d967609a92437938d7cbf22dd79a0192c37e: Status 404 returned error can't find the container with id cc04cdf178e15542b5e4238f4c52d967609a92437938d7cbf22dd79a0192c37e
Jul 14 17:38:05.620938 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:05.620921    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 17:38:06.056061 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:06.056026    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt" event={"ID":"dae2cca9-3da7-426f-887f-1e4c3e6cda8c","Type":"ContainerStarted","Data":"cc04cdf178e15542b5e4238f4c52d967609a92437938d7cbf22dd79a0192c37e"}
Jul 14 17:38:07.061566 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:07.061481    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt" event={"ID":"dae2cca9-3da7-426f-887f-1e4c3e6cda8c","Type":"ContainerStarted","Data":"c7471bb3c7c88f083eca806aa02adc27d8ae6c79c06763ed262680b07e0b2c02"}
Jul 14 17:38:07.079625 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:07.079581    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt" podStartSLOduration=1.05133937 podStartE2EDuration="2.079565814s" totalImagesPullingTime="1.028226444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:38:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:38:07.07807242 +0000 UTC m=+733.622476233" watchObservedRunningTime="2026-07-14 17:38:07.079565814 +0000 UTC m=+733.623969609"
Jul 14 17:38:15.492239 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:15.492201    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:15.492710 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:15.492280    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:15.504931 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:15.504910    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:16.113676 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:16.113648    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-77czt"
Jul 14 17:38:26.688410 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.688369    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"]
Jul 14 17:38:26.691871 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.691851    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.694615 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.694596    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 14 17:38:26.703547 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.703520    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"]
Jul 14 17:38:26.765841 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.765811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.765956 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.765847    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt94x\" (UniqueName: \"kubernetes.io/projected/fe033f73-aa97-4050-bbca-0a0e5277eb6b-kube-api-access-mt94x\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.765956 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.765881    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.765956 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.765920    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.765956 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.765942    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe033f73-aa97-4050-bbca-0a0e5277eb6b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.766087 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.765959    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867233 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867109    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt94x\" (UniqueName: \"kubernetes.io/projected/fe033f73-aa97-4050-bbca-0a0e5277eb6b-kube-api-access-mt94x\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867233 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867301 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867239    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867301 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867273    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe033f73-aa97-4050-bbca-0a0e5277eb6b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867411 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867546 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867525    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867610 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867565    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.867610 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.867602    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.869567 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.869545    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe033f73-aa97-4050-bbca-0a0e5277eb6b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.869857 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.869835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe033f73-aa97-4050-bbca-0a0e5277eb6b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:26.874798 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:26.874776    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt94x\" (UniqueName: \"kubernetes.io/projected/fe033f73-aa97-4050-bbca-0a0e5277eb6b-kube-api-access-mt94x\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm\" (UID: \"fe033f73-aa97-4050-bbca-0a0e5277eb6b\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:27.009204 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:27.009116    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:27.138153 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:27.138125    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"]
Jul 14 17:38:27.140563 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:38:27.140536    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe033f73_aa97_4050_bbca_0a0e5277eb6b.slice/crio-df663f5af3bbe77c8d26c760997c4b2e7446be44955a4d45ca3e7e4d93043dd7 WatchSource:0}: Error finding container df663f5af3bbe77c8d26c760997c4b2e7446be44955a4d45ca3e7e4d93043dd7: Status 404 returned error can't find the container with id df663f5af3bbe77c8d26c760997c4b2e7446be44955a4d45ca3e7e4d93043dd7
Jul 14 17:38:27.151074 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:27.151050    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm" event={"ID":"fe033f73-aa97-4050-bbca-0a0e5277eb6b","Type":"ContainerStarted","Data":"df663f5af3bbe77c8d26c760997c4b2e7446be44955a4d45ca3e7e4d93043dd7"}
Jul 14 17:38:28.157251 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:28.157219    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm" event={"ID":"fe033f73-aa97-4050-bbca-0a0e5277eb6b","Type":"ContainerStarted","Data":"75e693b01b1cf485877d7bd599b4ab041b2f5322153d4c7d40e86760e16e80dd"}
Jul 14 17:38:28.175436 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:28.175387    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm" podStartSLOduration=1.9850025900000001 podStartE2EDuration="2.175371903s" totalImagesPullingTime="190.369313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:38:28.173897083 +0000 UTC m=+754.718300879" watchObservedRunningTime="2026-07-14 17:38:28.175371903 +0000 UTC m=+754.719775697"
Jul 14 17:38:37.009565 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:37.009532    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:37.010071 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:37.009597    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:37.022048 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:37.022025    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:38:37.206240 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:38:37.206210    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm"
Jul 14 17:39:27.670220 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.670130    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cdf9fb9f4-j9zlc"]
Jul 14 17:39:27.673543 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.673526    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.680896 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.680871    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cdf9fb9f4-j9zlc"]
Jul 14 17:39:27.707765 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.707742    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177b6848-394f-4ade-885f-428922cd85d9-tls-cert\") pod \"authorino-cdf9fb9f4-j9zlc\" (UID: \"177b6848-394f-4ade-885f-428922cd85d9\") " pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.707896 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.707778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqlpw\" (UniqueName: \"kubernetes.io/projected/177b6848-394f-4ade-885f-428922cd85d9-kube-api-access-mqlpw\") pod \"authorino-cdf9fb9f4-j9zlc\" (UID: \"177b6848-394f-4ade-885f-428922cd85d9\") " pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.808671 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.808647    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177b6848-394f-4ade-885f-428922cd85d9-tls-cert\") pod \"authorino-cdf9fb9f4-j9zlc\" (UID: \"177b6848-394f-4ade-885f-428922cd85d9\") " pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.808793 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.808679    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqlpw\" (UniqueName: \"kubernetes.io/projected/177b6848-394f-4ade-885f-428922cd85d9-kube-api-access-mqlpw\") pod \"authorino-cdf9fb9f4-j9zlc\" (UID: \"177b6848-394f-4ade-885f-428922cd85d9\") " pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.811078 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.811053    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/177b6848-394f-4ade-885f-428922cd85d9-tls-cert\") pod \"authorino-cdf9fb9f4-j9zlc\" (UID: \"177b6848-394f-4ade-885f-428922cd85d9\") " pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.816184 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.816158    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqlpw\" (UniqueName: \"kubernetes.io/projected/177b6848-394f-4ade-885f-428922cd85d9-kube-api-access-mqlpw\") pod \"authorino-cdf9fb9f4-j9zlc\" (UID: \"177b6848-394f-4ade-885f-428922cd85d9\") " pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:27.983864 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:27.983778    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc"
Jul 14 17:39:28.112370 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:28.112331    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cdf9fb9f4-j9zlc"]
Jul 14 17:39:28.114493 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:39:28.114468    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod177b6848_394f_4ade_885f_428922cd85d9.slice/crio-afab63ad56de50a00fceed8e5e11af3742f3f4586113308cfae49ca3bdaaeec7 WatchSource:0}: Error finding container afab63ad56de50a00fceed8e5e11af3742f3f4586113308cfae49ca3bdaaeec7: Status 404 returned error can't find the container with id afab63ad56de50a00fceed8e5e11af3742f3f4586113308cfae49ca3bdaaeec7
Jul 14 17:39:28.429473 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:28.429446    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc" event={"ID":"177b6848-394f-4ade-885f-428922cd85d9","Type":"ContainerStarted","Data":"afab63ad56de50a00fceed8e5e11af3742f3f4586113308cfae49ca3bdaaeec7"}
Jul 14 17:39:29.438030 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.437985    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc" event={"ID":"177b6848-394f-4ade-885f-428922cd85d9","Type":"ContainerStarted","Data":"5235b0b6e3ea1fac3b1f336787407952cbd1a0443702a51a8faf8a96b3ec7016"}
Jul 14 17:39:29.456226 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.456159    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cdf9fb9f4-j9zlc" podStartSLOduration=2.156969979 podStartE2EDuration="2.456138789s" totalImagesPullingTime="299.16881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:39:29.455691899 +0000 UTC m=+816.000095693" watchObservedRunningTime="2026-07-14 17:39:29.456138789 +0000 UTC m=+816.000542584"
Jul 14 17:39:29.485976 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.485940    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59d487b9c5-gls2f"]
Jul 14 17:39:29.486223 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.486194    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59d487b9c5-gls2f" podUID="b749ffae-edad-4915-9dc8-ec15c63b596b" containerName="authorino" containerID="cri-o://9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89" gracePeriod=30
Jul 14 17:39:29.741814 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.741788    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:39:29.824144 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.824111    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b749ffae-edad-4915-9dc8-ec15c63b596b-tls-cert\") pod \"b749ffae-edad-4915-9dc8-ec15c63b596b\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") "
Jul 14 17:39:29.833976 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.833945    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b749ffae-edad-4915-9dc8-ec15c63b596b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b749ffae-edad-4915-9dc8-ec15c63b596b" (UID: "b749ffae-edad-4915-9dc8-ec15c63b596b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 17:39:29.924924 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.924882    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwwlc\" (UniqueName: \"kubernetes.io/projected/b749ffae-edad-4915-9dc8-ec15c63b596b-kube-api-access-bwwlc\") pod \"b749ffae-edad-4915-9dc8-ec15c63b596b\" (UID: \"b749ffae-edad-4915-9dc8-ec15c63b596b\") "
Jul 14 17:39:29.925213 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.925196    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b749ffae-edad-4915-9dc8-ec15c63b596b-tls-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:39:29.926899 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:29.926871    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b749ffae-edad-4915-9dc8-ec15c63b596b-kube-api-access-bwwlc" (OuterVolumeSpecName: "kube-api-access-bwwlc") pod "b749ffae-edad-4915-9dc8-ec15c63b596b" (UID: "b749ffae-edad-4915-9dc8-ec15c63b596b"). InnerVolumeSpecName "kube-api-access-bwwlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:39:30.025801 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.025725    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwwlc\" (UniqueName: \"kubernetes.io/projected/b749ffae-edad-4915-9dc8-ec15c63b596b-kube-api-access-bwwlc\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:39:30.444355 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.444312    2567 generic.go:356] "Generic (PLEG): container finished" podID="b749ffae-edad-4915-9dc8-ec15c63b596b" containerID="9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89" exitCode=0
Jul 14 17:39:30.444835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.444382    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59d487b9c5-gls2f"
Jul 14 17:39:30.444835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.444411    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59d487b9c5-gls2f" event={"ID":"b749ffae-edad-4915-9dc8-ec15c63b596b","Type":"ContainerDied","Data":"9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89"}
Jul 14 17:39:30.444835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.444473    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59d487b9c5-gls2f" event={"ID":"b749ffae-edad-4915-9dc8-ec15c63b596b","Type":"ContainerDied","Data":"003e89f6500fe1e21ddbfca36907204d692307d808a94c9eb4013590d21f3293"}
Jul 14 17:39:30.444835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.444505    2567 scope.go:117] "RemoveContainer" containerID="9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89"
Jul 14 17:39:30.453953 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.453930    2567 scope.go:117] "RemoveContainer" containerID="9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89"
Jul 14 17:39:30.454245 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:39:30.454215    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89\": container with ID starting with 9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89 not found: ID does not exist" containerID="9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89"
Jul 14 17:39:30.454332 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.454251    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89"} err="failed to get container status \"9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89\": rpc error: code = NotFound desc = could not find container \"9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89\": container with ID starting with 9cf4c58740024194498dba6152f254b845a281a2ec09cfbea2afb0452b0a6e89 not found: ID does not exist"
Jul 14 17:39:30.460658 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.460605    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59d487b9c5-gls2f"]
Jul 14 17:39:30.462741 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:30.462718    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59d487b9c5-gls2f"]
Jul 14 17:39:32.053665 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:39:32.053634    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b749ffae-edad-4915-9dc8-ec15c63b596b" path="/var/lib/kubelet/pods/b749ffae-edad-4915-9dc8-ec15c63b596b/volumes"
Jul 14 17:40:53.853564 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:40:53.853483    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:40:53.854315 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:40:53.854291    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:45:53.879641 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:45:53.879615    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:45:53.880560 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:45:53.880539    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:50:53.906293 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:50:53.906260    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:50:53.908338 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:50:53.908317    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:54:23.171658 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.171584    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"]
Jul 14 17:54:23.172030 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.171929    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b749ffae-edad-4915-9dc8-ec15c63b596b" containerName="authorino"
Jul 14 17:54:23.172030 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.171941    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b749ffae-edad-4915-9dc8-ec15c63b596b" containerName="authorino"
Jul 14 17:54:23.172030 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.172013    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b749ffae-edad-4915-9dc8-ec15c63b596b" containerName="authorino"
Jul 14 17:54:23.175204 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.175184    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.178290 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.178270    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7d767796-gw-openshift-default-dockercfg-zwvlz\""
Jul 14 17:54:23.194695 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.194650    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"]
Jul 14 17:54:23.228969 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.228928    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-socket\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.228988    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-token\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229042    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-credential-socket\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229084    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828zw\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-kube-api-access-828zw\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229125    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-podinfo\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229146    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-envoy\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229180 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229172    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istiod-ca-cert\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229527 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229195    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-data\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.229527 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.229224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-certs\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330019 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.329974    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-data\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330019 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330023    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-certs\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330254 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330072    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-socket\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330254 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330161    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-token\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330254 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330204    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-credential-socket\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330254 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330243    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828zw\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-kube-api-access-828zw\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330487 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-podinfo\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330487 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330315    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-envoy\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330487 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330363    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istiod-ca-cert\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330487 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330431    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-data\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330692 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330498    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-socket\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330785 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330752    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-credential-socket\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.330958 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330939    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istiod-ca-cert\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.331081 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.330943    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-certs\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.332806 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.332785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-envoy\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.333041 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.333021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-podinfo\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.346715 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.346688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-token\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.350962 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.350931    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828zw\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-kube-api-access-828zw\") pod \"e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") " pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.485849 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.485757    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:23.637467 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.637429    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"]
Jul 14 17:54:23.640729 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:54:23.640699    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdededc_2975_48f0_89b9_1cfe0396e9c9.slice/crio-9db728cdd92fc52a2c67b6d0468201d82c93a6289a9c98f43ff97a150003b96b WatchSource:0}: Error finding container 9db728cdd92fc52a2c67b6d0468201d82c93a6289a9c98f43ff97a150003b96b: Status 404 returned error can't find the container with id 9db728cdd92fc52a2c67b6d0468201d82c93a6289a9c98f43ff97a150003b96b
Jul 14 17:54:23.642852 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.642821    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:54:23.642946 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.642902    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:54:23.642946 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:23.642930    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:54:24.371300 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:24.371249    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" event={"ID":"7fdededc-2975-48f0-89b9-1cfe0396e9c9","Type":"ContainerStarted","Data":"ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"}
Jul 14 17:54:24.371791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:24.371379    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" event={"ID":"7fdededc-2975-48f0-89b9-1cfe0396e9c9","Type":"ContainerStarted","Data":"9db728cdd92fc52a2c67b6d0468201d82c93a6289a9c98f43ff97a150003b96b"}
Jul 14 17:54:24.400625 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:24.400575    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podStartSLOduration=1.40055561 podStartE2EDuration="1.40055561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:54:24.399195186 +0000 UTC m=+1710.943598983" watchObservedRunningTime="2026-07-14 17:54:24.40055561 +0000 UTC m=+1710.944959405"
Jul 14 17:54:24.486850 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:24.486808    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:24.491628 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:24.491603    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:25.377099 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:25.377060    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:25.378170 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:25.378147    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:41.418557 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:41.418512    2567 patch_prober.go:28] interesting pod/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 14 17:54:41.419033 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:41.418603    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 14 17:54:43.475921 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:43.475816    2567 generic.go:356] "Generic (PLEG): container finished" podID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerID="ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61" exitCode=0
Jul 14 17:54:43.475921 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:43.475885    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" event={"ID":"7fdededc-2975-48f0-89b9-1cfe0396e9c9","Type":"ContainerDied","Data":"ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"}
Jul 14 17:54:43.476398 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:43.476292    2567 scope.go:117] "RemoveContainer" containerID="ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"
Jul 14 17:54:43.477472 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:43.477443    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:54:43.477556 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:43.477510    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:54:43.477556 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:43.477541    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:54:44.482322 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:44.482279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" event={"ID":"7fdededc-2975-48f0-89b9-1cfe0396e9c9","Type":"ContainerStarted","Data":"5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6"}
Jul 14 17:54:44.486282 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:44.486260    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:44.486530 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:44.486507    2567 patch_prober.go:28] interesting pod/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 14 17:54:44.486627 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:44.486561    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 14 17:54:45.486527 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:45.486441    2567 patch_prober.go:28] interesting pod/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 14 17:54:45.486875 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:45.486514    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 14 17:54:46.486718 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:46.486683    2567 patch_prober.go:28] interesting pod/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 14 17:54:46.487121 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:46.486737    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 14 17:54:47.487226 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:47.487186    2567 patch_prober.go:28] interesting pod/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 14 17:54:47.487650 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:47.487256    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 14 17:54:48.502779 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:48.502747    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:48.503197 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:48.503116    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:54:48.503988 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:54:48.503963    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:55:29.065193 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:29.065162    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"]
Jul 14 17:55:29.065799 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:29.065468    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" containerID="cri-o://5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6" gracePeriod=30
Jul 14 17:55:33.505405 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:33.505367    2567 patch_prober.go:28] interesting pod/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 17:55:33.505766 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:33.505436    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 17:55:34.313560 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.313539    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:55:34.472049 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472023    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istiod-ca-cert\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472067    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-token\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472086    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-podinfo\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472106    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-socket\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472137    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-828zw\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-kube-api-access-828zw\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472155    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-certs\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472177    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-credential-socket\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472207 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472202    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-envoy\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472271    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-data\") pod \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\" (UID: \"7fdededc-2975-48f0-89b9-1cfe0396e9c9\") "
Jul 14 17:55:34.472767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472463    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:55:34.472767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472499    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:55:34.472767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472691    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:55:34.472767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472730    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:55:34.472767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472757    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istiod-ca-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.473070 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.472777    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.473559 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.473534    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:55:34.474460 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.474424    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:55:34.474570 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.474498    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:55:34.474681 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.474651    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-kube-api-access-828zw" (OuterVolumeSpecName: "kube-api-access-828zw") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "kube-api-access-828zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:55:34.474776 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.474681    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fdededc-2975-48f0-89b9-1cfe0396e9c9" (UID: "7fdededc-2975-48f0-89b9-1cfe0396e9c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 17:55:34.573843 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573819    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-token\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.573843 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573840    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-podinfo\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.574146 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573848    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-workload-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.574146 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573857    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828zw\" (UniqueName: \"kubernetes.io/projected/7fdededc-2975-48f0-89b9-1cfe0396e9c9-kube-api-access-828zw\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.574146 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573866    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-credential-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.574146 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573874    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-envoy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.574146 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.573882    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fdededc-2975-48f0-89b9-1cfe0396e9c9-istio-data\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:55:34.701791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.701714    2567 generic.go:356] "Generic (PLEG): container finished" podID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerID="5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6" exitCode=0
Jul 14 17:55:34.701791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.701757    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" event={"ID":"7fdededc-2975-48f0-89b9-1cfe0396e9c9","Type":"ContainerDied","Data":"5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6"}
Jul 14 17:55:34.701791 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.701763    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"
Jul 14 17:55:34.701976 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.701804    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z" event={"ID":"7fdededc-2975-48f0-89b9-1cfe0396e9c9","Type":"ContainerDied","Data":"9db728cdd92fc52a2c67b6d0468201d82c93a6289a9c98f43ff97a150003b96b"}
Jul 14 17:55:34.701976 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.701830    2567 scope.go:117] "RemoveContainer" containerID="5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6"
Jul 14 17:55:34.710767 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.710746    2567 scope.go:117] "RemoveContainer" containerID="ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"
Jul 14 17:55:34.718373 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.718355    2567 scope.go:117] "RemoveContainer" containerID="5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6"
Jul 14 17:55:34.718615 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:55:34.718596    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6\": container with ID starting with 5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6 not found: ID does not exist" containerID="5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6"
Jul 14 17:55:34.718676 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.718625    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6"} err="failed to get container status \"5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6\": rpc error: code = NotFound desc = could not find container \"5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6\": container with ID starting with 5c32efc9d1b5843d20ab94e123db793395c02c5d9b36704b8587d92cfb8226a6 not found: ID does not exist"
Jul 14 17:55:34.718676 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.718642    2567 scope.go:117] "RemoveContainer" containerID="ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"
Jul 14 17:55:34.718883 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:55:34.718864    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61\": container with ID starting with ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61 not found: ID does not exist" containerID="ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"
Jul 14 17:55:34.718927 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.718889    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61"} err="failed to get container status \"ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61\": rpc error: code = NotFound desc = could not find container \"ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61\": container with ID starting with ac1fe366cf0cf761f74df8e1acd24e66a097b1195d625bc0aba3bc0b90eeab61 not found: ID does not exist"
Jul 14 17:55:34.729983 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.729961    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"]
Jul 14 17:55:34.732578 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:34.732560    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7d767796-gw-openshift-default-7b6d4d9cd7-j2m4z"]
Jul 14 17:55:36.051074 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:36.051042    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" path="/var/lib/kubelet/pods/7fdededc-2975-48f0-89b9-1cfe0396e9c9/volumes"
Jul 14 17:55:41.102975 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.102942    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"]
Jul 14 17:55:41.103460 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.103293    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy"
Jul 14 17:55:41.103460 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.103307    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy"
Jul 14 17:55:41.103460 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.103429    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy"
Jul 14 17:55:41.103460 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.103443    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy"
Jul 14 17:55:41.108404 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.108380    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.116245 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.116206    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-efd93356-openshift-default-dockercfg-v975l\""
Jul 14 17:55:41.128162 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.128137    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"]
Jul 14 17:55:41.230294 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230231    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0665de1f-17da-4576-baa1-2d1c27701283-istiod-ca-cert\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230541 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230508    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-data\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230625 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230550    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rxkx\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-kube-api-access-6rxkx\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230625 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-credential-socket\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230739 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230634    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-certs\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230739 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-socket\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230739 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-envoy\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230738    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0665de1f-17da-4576-baa1-2d1c27701283-istio-podinfo\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.230891 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.230762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-istio-token\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332245 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332212    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0665de1f-17da-4576-baa1-2d1c27701283-istiod-ca-cert\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332245 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332247    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-data\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332263    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rxkx\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-kube-api-access-6rxkx\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332390    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-credential-socket\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332438 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332420    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-certs\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332593 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332441    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-socket\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332593 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-envoy\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332593 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332511    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0665de1f-17da-4576-baa1-2d1c27701283-istio-podinfo\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332593 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332535    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-istio-token\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332797 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332622    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-data\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332797 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332743    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-credential-socket\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332898 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332818    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-socket\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.332952 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.332922    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-certs\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.333117 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.333098    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0665de1f-17da-4576-baa1-2d1c27701283-istiod-ca-cert\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.334603 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.334584    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-envoy\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.334911 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.334892    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0665de1f-17da-4576-baa1-2d1c27701283-istio-podinfo\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.342751 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.342731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-istio-token\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.343234 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.343215    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rxkx\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-kube-api-access-6rxkx\") pod \"e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") " pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.419867 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.419814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:41.567829 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.567804    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"]
Jul 14 17:55:41.567935 ip-10-0-138-98 kubenswrapper[2567]: W0714 17:55:41.567913    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0665de1f_17da_4576_baa1_2d1c27701283.slice/crio-60addeee7ba05e87b6063120afabf34e82d4f0fbc4ad44250f93b805cdce6e01 WatchSource:0}: Error finding container 60addeee7ba05e87b6063120afabf34e82d4f0fbc4ad44250f93b805cdce6e01: Status 404 returned error can't find the container with id 60addeee7ba05e87b6063120afabf34e82d4f0fbc4ad44250f93b805cdce6e01
Jul 14 17:55:41.570158 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.570128    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:55:41.570224 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.570188    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:55:41.570224 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.570219    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:55:41.735942 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.735862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" event={"ID":"0665de1f-17da-4576-baa1-2d1c27701283","Type":"ContainerStarted","Data":"b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"}
Jul 14 17:55:41.735942 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.735926    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" event={"ID":"0665de1f-17da-4576-baa1-2d1c27701283","Type":"ContainerStarted","Data":"60addeee7ba05e87b6063120afabf34e82d4f0fbc4ad44250f93b805cdce6e01"}
Jul 14 17:55:41.766825 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:41.766779    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" podStartSLOduration=0.766764774 podStartE2EDuration="766.764774ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 17:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 17:55:41.764600811 +0000 UTC m=+1788.309004631" watchObservedRunningTime="2026-07-14 17:55:41.766764774 +0000 UTC m=+1788.311168569"
Jul 14 17:55:42.420176 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:42.420139    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:42.425103 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:42.425080    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:42.741073 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:42.740996    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:42.742049 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:42.742031    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:53.793190 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.793158    2567 generic.go:356] "Generic (PLEG): container finished" podID="0665de1f-17da-4576-baa1-2d1c27701283" containerID="b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1" exitCode=0
Jul 14 17:55:53.793605 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.793194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" event={"ID":"0665de1f-17da-4576-baa1-2d1c27701283","Type":"ContainerDied","Data":"b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"}
Jul 14 17:55:53.793605 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.793559    2567 scope.go:117] "RemoveContainer" containerID="b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"
Jul 14 17:55:53.794579 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.794548    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:55:53.794649 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.794624    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:55:53.794685 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.794663    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 17:55:53.937509 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.937482    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:55:53.940878 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:53.940854    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 17:55:54.799380 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:54.799323    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" event={"ID":"0665de1f-17da-4576-baa1-2d1c27701283","Type":"ContainerStarted","Data":"7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45"}
Jul 14 17:55:55.420505 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:55.420473    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:55.421514 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:55.421489    2567 patch_prober.go:28] interesting pod/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 14 17:55:55.421694 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:55.421547    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 14 17:55:56.420654 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:56.420623    2567 patch_prober.go:28] interesting pod/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 14 17:55:56.420996 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:56.420675    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 14 17:55:57.421115 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:57.421077    2567 patch_prober.go:28] interesting pod/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 14 17:55:57.421507 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:57.421143    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 14 17:55:57.742486 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:57.741610    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:58.421110 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:58.421070    2567 patch_prober.go:28] interesting pod/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Jul 14 17:55:58.421283 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:58.421137    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Jul 14 17:55:59.424507 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:59.424481    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:55:59.425674 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:55:59.425651    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:57:03.493624 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:03.493587    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"]
Jul 14 17:57:03.494085 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:03.493774    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy" containerID="cri-o://7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45" gracePeriod=30
Jul 14 17:57:08.734036 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.734008    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:57:08.852286 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852257    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0665de1f-17da-4576-baa1-2d1c27701283-istio-podinfo\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852449 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852305    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-certs\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852449 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852325    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-data\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852449 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852361    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-istio-token\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852449 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852406    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rxkx\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-kube-api-access-6rxkx\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852449 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852423    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-envoy\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852721 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852459    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-credential-socket\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852721 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852491    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0665de1f-17da-4576-baa1-2d1c27701283-istiod-ca-cert\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852721 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852520    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-socket\") pod \"0665de1f-17da-4576-baa1-2d1c27701283\" (UID: \"0665de1f-17da-4576-baa1-2d1c27701283\") "
Jul 14 17:57:08.852721 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852630    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:57:08.852921 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852845    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:57:08.852921 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852906    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.852921 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.852912    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:57:08.853165 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.853143    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0665de1f-17da-4576-baa1-2d1c27701283-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 17:57:08.853322 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.853300    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-data" (OuterVolumeSpecName: "istio-data") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:57:08.854587 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.854555    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 17:57:08.854587 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.854569    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-kube-api-access-6rxkx" (OuterVolumeSpecName: "kube-api-access-6rxkx") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "kube-api-access-6rxkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:57:08.854796 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.854775    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0665de1f-17da-4576-baa1-2d1c27701283-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 17:57:08.854849 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.854791    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-istio-token" (OuterVolumeSpecName: "istio-token") pod "0665de1f-17da-4576-baa1-2d1c27701283" (UID: "0665de1f-17da-4576-baa1-2d1c27701283"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 17:57:08.953661 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953635    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0665de1f-17da-4576-baa1-2d1c27701283-istio-podinfo\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953661 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953657    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-data\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953661 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953666    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-istio-token\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953675    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rxkx\" (UniqueName: \"kubernetes.io/projected/0665de1f-17da-4576-baa1-2d1c27701283-kube-api-access-6rxkx\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953684    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-istio-envoy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953692    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-credential-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953700    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0665de1f-17da-4576-baa1-2d1c27701283-istiod-ca-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:08.953835 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:08.953708    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0665de1f-17da-4576-baa1-2d1c27701283-workload-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 17:57:09.117774 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.117741    2567 generic.go:356] "Generic (PLEG): container finished" podID="0665de1f-17da-4576-baa1-2d1c27701283" containerID="7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45" exitCode=0
Jul 14 17:57:09.117928 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.117779    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"
Jul 14 17:57:09.117928 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.117822    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" event={"ID":"0665de1f-17da-4576-baa1-2d1c27701283","Type":"ContainerDied","Data":"7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45"}
Jul 14 17:57:09.117928 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.117871    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c" event={"ID":"0665de1f-17da-4576-baa1-2d1c27701283","Type":"ContainerDied","Data":"60addeee7ba05e87b6063120afabf34e82d4f0fbc4ad44250f93b805cdce6e01"}
Jul 14 17:57:09.117928 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.117892    2567 scope.go:117] "RemoveContainer" containerID="7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45"
Jul 14 17:57:09.126641 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.126628    2567 scope.go:117] "RemoveContainer" containerID="b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"
Jul 14 17:57:09.133897 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.133879    2567 scope.go:117] "RemoveContainer" containerID="7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45"
Jul 14 17:57:09.134132 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:57:09.134115    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45\": container with ID starting with 7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45 not found: ID does not exist" containerID="7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45"
Jul 14 17:57:09.134193 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.134138    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45"} err="failed to get container status \"7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45\": rpc error: code = NotFound desc = could not find container \"7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45\": container with ID starting with 7f2d48fc7b34004336c3949ce806e07c4d0874b417aa0b2ced167eb3c4686c45 not found: ID does not exist"
Jul 14 17:57:09.134193 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.134153    2567 scope.go:117] "RemoveContainer" containerID="b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"
Jul 14 17:57:09.134371 ip-10-0-138-98 kubenswrapper[2567]: E0714 17:57:09.134333    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1\": container with ID starting with b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1 not found: ID does not exist" containerID="b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"
Jul 14 17:57:09.134415 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.134377    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1"} err="failed to get container status \"b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1\": rpc error: code = NotFound desc = could not find container \"b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1\": container with ID starting with b9fe3e1d0912f77188bb3664f06d25cf0d9049cd1d78df6403bfe1ff954ccab1 not found: ID does not exist"
Jul 14 17:57:09.140013 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.139991    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"]
Jul 14 17:57:09.143490 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:09.143471    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-efd93356-openshift-default-5f8ddbbb77-7zl8c"]
Jul 14 17:57:10.050769 ip-10-0-138-98 kubenswrapper[2567]: I0714 17:57:10.050730    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0665de1f-17da-4576-baa1-2d1c27701283" path="/var/lib/kubelet/pods/0665de1f-17da-4576-baa1-2d1c27701283/volumes"
Jul 14 18:00:53.964910 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:00:53.964867    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 18:00:53.971750 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:00:53.971730    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 18:01:05.820737 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.820706    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"]
Jul 14 18:01:05.821225 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.821206    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy"
Jul 14 18:01:05.821304 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.821228    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy"
Jul 14 18:01:05.821304 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.821266    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy"
Jul 14 18:01:05.821304 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.821275    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fdededc-2975-48f0-89b9-1cfe0396e9c9" containerName="istio-proxy"
Jul 14 18:01:05.821491 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.821393    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy"
Jul 14 18:01:05.824213 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.824191    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.827216 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.827194    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3d8a768b-openshift-default-dockercfg-nzpf4\""
Jul 14 18:01:05.847008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.846984    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"]
Jul 14 18:01:05.976832 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.976798    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-data\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977009 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.976838    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-envoy\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977009 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.976892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-credential-socket\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977009 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.976998    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5llb\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-kube-api-access-x5llb\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977159 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.977077    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-token\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977159 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.977098    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istiod-ca-cert\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977159 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.977120    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-certs\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977159 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.977157    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-socket\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:05.977298 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:05.977181    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-podinfo\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078292 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078196    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-token\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078292 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078247    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istiod-ca-cert\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078561 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078293    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-certs\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078561 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078371    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-socket\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078561 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078418    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-podinfo\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078561 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-data\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078772 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-envoy\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078772 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-credential-socket\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078772 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078687    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5llb\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-kube-api-access-x5llb\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.078929 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.078849    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-socket\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.079152 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.079122    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-data\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.079152 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.079140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istiod-ca-cert\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.079338 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.079173    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-certs\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.079338 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.079199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-credential-socket\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.080842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.080822    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-envoy\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.081540 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.081519    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-podinfo\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.087099 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.087073    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-token\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.087337 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.087318    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5llb\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-kube-api-access-x5llb\") pod \"e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") " pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.134968 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.134943    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:06.276127 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.276102    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"]
Jul 14 18:01:06.278631 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:01:06.278605    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea60e9b_40d2_4cb1_a704_959015a1cf1b.slice/crio-3855000c2449c74446fd4814bb3009f4b128dcb4c7de2fa40007d26701f87f87 WatchSource:0}: Error finding container 3855000c2449c74446fd4814bb3009f4b128dcb4c7de2fa40007d26701f87f87: Status 404 returned error can't find the container with id 3855000c2449c74446fd4814bb3009f4b128dcb4c7de2fa40007d26701f87f87
Jul 14 18:01:06.281283 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.281240    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:01:06.281389 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.281328    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:01:06.281443 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:06.281390    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:01:07.160573 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:07.160540    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" event={"ID":"cea60e9b-40d2-4cb1-a704-959015a1cf1b","Type":"ContainerStarted","Data":"673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"}
Jul 14 18:01:07.160915 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:07.160589    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" event={"ID":"cea60e9b-40d2-4cb1-a704-959015a1cf1b","Type":"ContainerStarted","Data":"3855000c2449c74446fd4814bb3009f4b128dcb4c7de2fa40007d26701f87f87"}
Jul 14 18:01:07.182971 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:07.182923    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podStartSLOduration=2.182910247 podStartE2EDuration="2.182910247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:01:07.181605629 +0000 UTC m=+2113.726009491" watchObservedRunningTime="2026-07-14 18:01:07.182910247 +0000 UTC m=+2113.727314042"
Jul 14 18:01:08.135859 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:08.135807    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:08.141317 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:08.141288    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:08.166194 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:08.166161    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:08.167144 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:08.167125    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:22.238551 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:22.238517    2567 generic.go:356] "Generic (PLEG): container finished" podID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerID="673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909" exitCode=0
Jul 14 18:01:22.238954 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:22.238588    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" event={"ID":"cea60e9b-40d2-4cb1-a704-959015a1cf1b","Type":"ContainerDied","Data":"673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"}
Jul 14 18:01:22.238999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:22.238976    2567 scope.go:117] "RemoveContainer" containerID="673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"
Jul 14 18:01:22.239898 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:22.239873    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:01:22.239955 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:22.239939    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:01:22.239994 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:22.239983    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:01:23.166432 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:23.166381    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:23.247258 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:23.247214    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" event={"ID":"cea60e9b-40d2-4cb1-a704-959015a1cf1b","Type":"ContainerStarted","Data":"637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6"}
Jul 14 18:01:24.136289 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:24.136254    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:24.137365 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:24.137315    2567 patch_prober.go:28] interesting pod/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 14 18:01:24.137506 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:24.137389    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 14 18:01:25.135427 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:25.135388    2567 patch_prober.go:28] interesting pod/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 14 18:01:25.136006 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:25.135972    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 14 18:01:26.136213 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:26.136168    2567 patch_prober.go:28] interesting pod/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 14 18:01:26.136729 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:26.136244    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 14 18:01:27.135817 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:27.135775    2567 patch_prober.go:28] interesting pod/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Jul 14 18:01:27.136034 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:27.135848    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Jul 14 18:01:28.139487 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:28.139458    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:28.139927 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:28.139798    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:28.140408 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:28.140391    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:01:30.086213 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.086176    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"]
Jul 14 18:01:30.086746 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.086726    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy"
Jul 14 18:01:30.086798 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.086753    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy"
Jul 14 18:01:30.086855 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.086843    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0665de1f-17da-4576-baa1-2d1c27701283" containerName="istio-proxy"
Jul 14 18:01:30.089959 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.089937    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.093949 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.093916    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d8a768b\"/\"e2e-lifecycle-model-3d8a768b-kserve-self-signed-certs\""
Jul 14 18:01:30.094084 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.093984    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d8a768b\"/\"openshift-service-ca.crt\""
Jul 14 18:01:30.094084 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.094046    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d8a768b\"/\"default-dockercfg-gb4p6\""
Jul 14 18:01:30.094250 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.094107    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d8a768b\"/\"kube-root-ca.crt\""
Jul 14 18:01:30.099748 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.099728    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"]
Jul 14 18:01:30.195446 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.195422    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-dshm\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.195587 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.195481    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-home\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.195587 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.195503    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfjzt\" (UniqueName: \"kubernetes.io/projected/3f145c81-7937-4820-983f-45f61b291c0a-kube-api-access-dfjzt\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.195587 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.195545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-tmp-dir\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.195587 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.195572    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-model-cache\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.195729 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.195599    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f145c81-7937-4820-983f-45f61b291c0a-tls-certs\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.296627 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.296598    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f145c81-7937-4820-983f-45f61b291c0a-tls-certs\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.296774 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.296661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-dshm\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.296774 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.296712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-home\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.296774 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.296740    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfjzt\" (UniqueName: \"kubernetes.io/projected/3f145c81-7937-4820-983f-45f61b291c0a-kube-api-access-dfjzt\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.296900 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.296771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-tmp-dir\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.296900 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.296793    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-model-cache\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.297157 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.297132    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-home\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.297157 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.297150    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-tmp-dir\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.297323 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.297202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-model-cache\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.299016 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.298991    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-dshm\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.299203 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.299189    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f145c81-7937-4820-983f-45f61b291c0a-tls-certs\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.305639 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.305617    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfjzt\" (UniqueName: \"kubernetes.io/projected/3f145c81-7937-4820-983f-45f61b291c0a-kube-api-access-dfjzt\") pod \"e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") " pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.399625 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.399563    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:30.529315 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:30.529285    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"]
Jul 14 18:01:30.530465 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:01:30.530428    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f145c81_7937_4820_983f_45f61b291c0a.slice/crio-8efa39bba110ac86c95fd45e3167ab55a429ea5271d7aa11cfbf8d9a8275f872 WatchSource:0}: Error finding container 8efa39bba110ac86c95fd45e3167ab55a429ea5271d7aa11cfbf8d9a8275f872: Status 404 returned error can't find the container with id 8efa39bba110ac86c95fd45e3167ab55a429ea5271d7aa11cfbf8d9a8275f872
Jul 14 18:01:31.291125 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:31.291087    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499" event={"ID":"3f145c81-7937-4820-983f-45f61b291c0a","Type":"ContainerStarted","Data":"3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf"}
Jul 14 18:01:31.291600 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:31.291134    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499" event={"ID":"3f145c81-7937-4820-983f-45f61b291c0a","Type":"ContainerStarted","Data":"8efa39bba110ac86c95fd45e3167ab55a429ea5271d7aa11cfbf8d9a8275f872"}
Jul 14 18:01:31.310180 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:31.310137    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499" podStartSLOduration=1.310124323 podStartE2EDuration="1.310124323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:01:31.308610133 +0000 UTC m=+2137.853013927" watchObservedRunningTime="2026-07-14 18:01:31.310124323 +0000 UTC m=+2137.854528118"
Jul 14 18:01:40.400018 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:40.399982    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:40.400533 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:40.400277    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:40.412337 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:40.412316    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:01:41.347978 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:01:41.347951    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:02:11.943325 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:11.943247    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"]
Jul 14 18:02:26.963461 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:26.963334    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499" podUID="3f145c81-7937-4820-983f-45f61b291c0a" containerName="main" containerID="cri-o://3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf" gracePeriod=45
Jul 14 18:02:27.212185 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.212161    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:02:27.296566 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296505    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-home\") pod \"3f145c81-7937-4820-983f-45f61b291c0a\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") "
Jul 14 18:02:27.296566 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296566    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-tmp-dir\") pod \"3f145c81-7937-4820-983f-45f61b291c0a\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") "
Jul 14 18:02:27.296762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296583    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-model-cache\") pod \"3f145c81-7937-4820-983f-45f61b291c0a\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") "
Jul 14 18:02:27.296762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296615    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f145c81-7937-4820-983f-45f61b291c0a-tls-certs\") pod \"3f145c81-7937-4820-983f-45f61b291c0a\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") "
Jul 14 18:02:27.296762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296645    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-dshm\") pod \"3f145c81-7937-4820-983f-45f61b291c0a\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") "
Jul 14 18:02:27.296762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296663    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfjzt\" (UniqueName: \"kubernetes.io/projected/3f145c81-7937-4820-983f-45f61b291c0a-kube-api-access-dfjzt\") pod \"3f145c81-7937-4820-983f-45f61b291c0a\" (UID: \"3f145c81-7937-4820-983f-45f61b291c0a\") "
Jul 14 18:02:27.296945 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296795    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-home" (OuterVolumeSpecName: "home") pod "3f145c81-7937-4820-983f-45f61b291c0a" (UID: "3f145c81-7937-4820-983f-45f61b291c0a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:02:27.297084 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.297056    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f145c81-7937-4820-983f-45f61b291c0a" (UID: "3f145c81-7937-4820-983f-45f61b291c0a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:02:27.297144 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.296837    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f145c81-7937-4820-983f-45f61b291c0a" (UID: "3f145c81-7937-4820-983f-45f61b291c0a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:02:27.301390 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.299322    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f145c81-7937-4820-983f-45f61b291c0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f145c81-7937-4820-983f-45f61b291c0a" (UID: "3f145c81-7937-4820-983f-45f61b291c0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:02:27.301390 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.299463    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f145c81-7937-4820-983f-45f61b291c0a-kube-api-access-dfjzt" (OuterVolumeSpecName: "kube-api-access-dfjzt") pod "3f145c81-7937-4820-983f-45f61b291c0a" (UID: "3f145c81-7937-4820-983f-45f61b291c0a"). InnerVolumeSpecName "kube-api-access-dfjzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:02:27.304465 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.304438    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-dshm" (OuterVolumeSpecName: "dshm") pod "3f145c81-7937-4820-983f-45f61b291c0a" (UID: "3f145c81-7937-4820-983f-45f61b291c0a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:02:27.398540 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.398520    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-dshm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:02:27.398637 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.398557    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfjzt\" (UniqueName: \"kubernetes.io/projected/3f145c81-7937-4820-983f-45f61b291c0a-kube-api-access-dfjzt\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:02:27.398637 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.398571    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-home\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:02:27.398637 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.398583    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-tmp-dir\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:02:27.398637 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.398592    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f145c81-7937-4820-983f-45f61b291c0a-model-cache\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:02:27.398637 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.398600    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f145c81-7937-4820-983f-45f61b291c0a-tls-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:02:27.546503 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.546476    2567 generic.go:356] "Generic (PLEG): container finished" podID="3f145c81-7937-4820-983f-45f61b291c0a" containerID="3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf" exitCode=0
Jul 14 18:02:27.546622 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.546531    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"
Jul 14 18:02:27.546622 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.546555    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499" event={"ID":"3f145c81-7937-4820-983f-45f61b291c0a","Type":"ContainerDied","Data":"3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf"}
Jul 14 18:02:27.546622 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.546593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499" event={"ID":"3f145c81-7937-4820-983f-45f61b291c0a","Type":"ContainerDied","Data":"8efa39bba110ac86c95fd45e3167ab55a429ea5271d7aa11cfbf8d9a8275f872"}
Jul 14 18:02:27.546622 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.546610    2567 scope.go:117] "RemoveContainer" containerID="3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf"
Jul 14 18:02:27.555022 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.555007    2567 scope.go:117] "RemoveContainer" containerID="3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf"
Jul 14 18:02:27.555283 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:02:27.555265    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf\": container with ID starting with 3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf not found: ID does not exist" containerID="3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf"
Jul 14 18:02:27.555337 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.555290    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf"} err="failed to get container status \"3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf\": rpc error: code = NotFound desc = could not find container \"3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf\": container with ID starting with 3d523d58e7b15068008fd8f986f556beb47d7244d87c078f7ce645c9f6dc6faf not found: ID does not exist"
Jul 14 18:02:27.568235 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.568211    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"]
Jul 14 18:02:27.574942 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:27.574921    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d8a768b/e2e-lifecycle-model-3d8a768b-kserve-5bb5b6c59-hq499"]
Jul 14 18:02:28.051190 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:02:28.051163    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f145c81-7937-4820-983f-45f61b291c0a" path="/var/lib/kubelet/pods/3f145c81-7937-4820-983f-45f61b291c0a/volumes"
Jul 14 18:03:09.304178 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:09.304146    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"]
Jul 14 18:03:09.304793 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:09.304337    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy" containerID="cri-o://637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6" gracePeriod=30
Jul 14 18:03:13.142853 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:13.142816    2567 patch_prober.go:28] interesting pod/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 18:03:13.143208 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:13.142879    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 18:03:14.561260 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.561230    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:03:14.681185 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681143    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-podinfo\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681393 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681273    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-data\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681393 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681302    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5llb\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-kube-api-access-x5llb\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681393 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681375    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-socket\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681586 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681481    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-certs\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681586 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681534    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istiod-ca-cert\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681699 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681585    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-credential-socket\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681699 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681608    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:03:14.681699 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681643    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-token\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681699 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681673    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-envoy\") pod \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\" (UID: \"cea60e9b-40d2-4cb1-a704-959015a1cf1b\") "
Jul 14 18:03:14.681896 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681707    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:03:14.681958 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:03:14.681958 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.681918    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:03:14.682063 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.682019    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.682063 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.682036    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-workload-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.682063 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.682049    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istiod-ca-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.682232 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.682063    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-credential-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.682290 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.682242    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:03:14.683606 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.683557    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 18:03:14.683930 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.683896    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-kube-api-access-x5llb" (OuterVolumeSpecName: "kube-api-access-x5llb") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "kube-api-access-x5llb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:03:14.684040 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.684017    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:03:14.684240 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.684222    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "cea60e9b-40d2-4cb1-a704-959015a1cf1b" (UID: "cea60e9b-40d2-4cb1-a704-959015a1cf1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:03:14.768146 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.768117    2567 generic.go:356] "Generic (PLEG): container finished" podID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerID="637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6" exitCode=0
Jul 14 18:03:14.768315 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.768160    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" event={"ID":"cea60e9b-40d2-4cb1-a704-959015a1cf1b","Type":"ContainerDied","Data":"637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6"}
Jul 14 18:03:14.768315 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.768179    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"
Jul 14 18:03:14.768315 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.768202    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899" event={"ID":"cea60e9b-40d2-4cb1-a704-959015a1cf1b","Type":"ContainerDied","Data":"3855000c2449c74446fd4814bb3009f4b128dcb4c7de2fa40007d26701f87f87"}
Jul 14 18:03:14.768315 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.768223    2567 scope.go:117] "RemoveContainer" containerID="637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6"
Jul 14 18:03:14.777605 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.777585    2567 scope.go:117] "RemoveContainer" containerID="673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"
Jul 14 18:03:14.783392 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.783373    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-data\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.783392 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.783392    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5llb\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-kube-api-access-x5llb\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.783533 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.783402    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-token\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.783533 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.783411    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-envoy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.783533 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.783418    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea60e9b-40d2-4cb1-a704-959015a1cf1b-istio-podinfo\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:03:14.785416 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.785398    2567 scope.go:117] "RemoveContainer" containerID="637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6"
Jul 14 18:03:14.785719 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:03:14.785700    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6\": container with ID starting with 637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6 not found: ID does not exist" containerID="637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6"
Jul 14 18:03:14.785776 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.785728    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6"} err="failed to get container status \"637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6\": rpc error: code = NotFound desc = could not find container \"637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6\": container with ID starting with 637ae4f67e26a15ed5320818b847294c8add2dfeb49dca8bd69b8a09d29a9cc6 not found: ID does not exist"
Jul 14 18:03:14.785776 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.785746    2567 scope.go:117] "RemoveContainer" containerID="673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"
Jul 14 18:03:14.785964 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:03:14.785945    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909\": container with ID starting with 673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909 not found: ID does not exist" containerID="673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"
Jul 14 18:03:14.786050 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.785961    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909"} err="failed to get container status \"673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909\": rpc error: code = NotFound desc = could not find container \"673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909\": container with ID starting with 673721af117ea4ef4fecee83bdc8de9376e9f5d82747f49709251fa7d04ed909 not found: ID does not exist"
Jul 14 18:03:14.791211 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.791188    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"]
Jul 14 18:03:14.797953 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:14.797929    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3d8a768b-openshift-default-84cdd8b55f-m9899"]
Jul 14 18:03:16.050444 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:03:16.050368    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" path="/var/lib/kubelet/pods/cea60e9b-40d2-4cb1-a704-959015a1cf1b/volumes"
Jul 14 18:04:12.186358 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186311    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"]
Jul 14 18:04:12.186857 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186831    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f145c81-7937-4820-983f-45f61b291c0a" containerName="main"
Jul 14 18:04:12.186857 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186857    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f145c81-7937-4820-983f-45f61b291c0a" containerName="main"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186868    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186877    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186889    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186894    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186953    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186962    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea60e9b-40d2-4cb1-a704-959015a1cf1b" containerName="istio-proxy"
Jul 14 18:04:12.187008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.186974    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f145c81-7937-4820-983f-45f61b291c0a" containerName="main"
Jul 14 18:04:12.189770 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.189751    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.192865 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.192847    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2d5136-openshift-default-dockercfg-r8dxb\""
Jul 14 18:04:12.203557 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.203534    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"]
Jul 14 18:04:12.219738 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219715    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-podinfo\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219824 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-certs\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219824 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219778    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-token\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219824 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219797    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd49a5a8-05df-4c25-856a-24f65dc4d491-istiod-ca-cert\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219824 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219822    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-socket\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219973 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219851    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-envoy\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219973 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-credential-socket\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219973 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219907    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-data\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.219973 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.219945    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599t9\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-kube-api-access-599t9\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321260 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321233    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-podinfo\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321391 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-certs\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321391 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321303    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-token\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321391 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321319    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd49a5a8-05df-4c25-856a-24f65dc4d491-istiod-ca-cert\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321391 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321361    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-socket\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321593 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321405    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-envoy\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321593 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-credential-socket\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321593 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-data\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321593 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599t9\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-kube-api-access-599t9\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321804 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321737    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-certs\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.321804 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.321791    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-socket\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.322054 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.322027    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-credential-socket\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.322158 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.322079    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-data\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.322158 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.322151    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd49a5a8-05df-4c25-856a-24f65dc4d491-istiod-ca-cert\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.323861 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.323835    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-podinfo\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.323954 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.323919    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-envoy\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.329991 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.329960    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-token\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.330756 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.330717    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-599t9\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-kube-api-access-599t9\") pod \"e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") " pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.499927 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.499859    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:12.642870 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.642846    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"]
Jul 14 18:04:12.643457 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:04:12.643433    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd49a5a8_05df_4c25_856a_24f65dc4d491.slice/crio-b7f94d3c2ecc903761e5ea9f923ff32ac2321bb618f90835f3b6293cf5847b50 WatchSource:0}: Error finding container b7f94d3c2ecc903761e5ea9f923ff32ac2321bb618f90835f3b6293cf5847b50: Status 404 returned error can't find the container with id b7f94d3c2ecc903761e5ea9f923ff32ac2321bb618f90835f3b6293cf5847b50
Jul 14 18:04:12.645751 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.645714    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:04:12.645860 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.645792    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:04:12.645860 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:12.645838    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:04:13.032062 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:13.032022    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" event={"ID":"cd49a5a8-05df-4c25-856a-24f65dc4d491","Type":"ContainerStarted","Data":"ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"}
Jul 14 18:04:13.032235 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:13.032084    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" event={"ID":"cd49a5a8-05df-4c25-856a-24f65dc4d491","Type":"ContainerStarted","Data":"b7f94d3c2ecc903761e5ea9f923ff32ac2321bb618f90835f3b6293cf5847b50"}
Jul 14 18:04:13.500242 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:13.500213    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:13.504812 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:13.504792    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:13.525972 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:13.525931    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podStartSLOduration=1.525917562 podStartE2EDuration="1.525917562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:04:13.057521261 +0000 UTC m=+2299.601925057" watchObservedRunningTime="2026-07-14 18:04:13.525917562 +0000 UTC m=+2300.070321356"
Jul 14 18:04:14.036405 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:14.036378    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:14.037268 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:14.037248    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:24.483426 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.483387    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"]
Jul 14 18:04:24.486401 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.486379    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.490555 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.490532    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ef174\"/\"openshift-service-ca.crt\""
Jul 14 18:04:24.490856 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.490537    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ef174\"/\"test-model-4ef174-kserve-self-signed-certs\""
Jul 14 18:04:24.490981 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.490598    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ef174\"/\"default-dockercfg-z68kt\""
Jul 14 18:04:24.491065 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.490609    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ef174\"/\"kube-root-ca.crt\""
Jul 14 18:04:24.496557 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.496536    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"]
Jul 14 18:04:24.518681 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.518655    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-tmp-dir\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.518771 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.518701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-home\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.518771 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.518719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-model-cache\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.518860 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.518811    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-dshm\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.518860 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.518843    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caf5a7-431a-4d4d-b770-cf110f2e7212-tls-certs\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.518945 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.518865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhwn\" (UniqueName: \"kubernetes.io/projected/99caf5a7-431a-4d4d-b770-cf110f2e7212-kube-api-access-xfhwn\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620011 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.619976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-home\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620011 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620010    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-model-cache\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620180 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-dshm\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620180 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620158    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caf5a7-431a-4d4d-b770-cf110f2e7212-tls-certs\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620278 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620207    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhwn\" (UniqueName: \"kubernetes.io/projected/99caf5a7-431a-4d4d-b770-cf110f2e7212-kube-api-access-xfhwn\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620339 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-tmp-dir\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620429 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620333    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-home\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620486 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620463    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-model-cache\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.620660 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.620641    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-tmp-dir\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.622169 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.622153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-dshm\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.622569 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.622555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caf5a7-431a-4d4d-b770-cf110f2e7212-tls-certs\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.630503 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.630452    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhwn\" (UniqueName: \"kubernetes.io/projected/99caf5a7-431a-4d4d-b770-cf110f2e7212-kube-api-access-xfhwn\") pod \"test-model-4ef174-kserve-d95898f9f-f6zth\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") " pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.796996 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.796918    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:24.973143 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:24.973064    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"]
Jul 14 18:04:24.975616 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:04:24.975587    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99caf5a7_431a_4d4d_b770_cf110f2e7212.slice/crio-c1232922ea34358c8906837733e6544f61b3fdd2c0828d8c9554096853e1b5aa WatchSource:0}: Error finding container c1232922ea34358c8906837733e6544f61b3fdd2c0828d8c9554096853e1b5aa: Status 404 returned error can't find the container with id c1232922ea34358c8906837733e6544f61b3fdd2c0828d8c9554096853e1b5aa
Jul 14 18:04:25.084248 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:25.084213    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth" event={"ID":"99caf5a7-431a-4d4d-b770-cf110f2e7212","Type":"ContainerStarted","Data":"775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b"}
Jul 14 18:04:25.084377 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:25.084256    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth" event={"ID":"99caf5a7-431a-4d4d-b770-cf110f2e7212","Type":"ContainerStarted","Data":"c1232922ea34358c8906837733e6544f61b3fdd2c0828d8c9554096853e1b5aa"}
Jul 14 18:04:25.102393 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:25.102327    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth" podStartSLOduration=1.102312218 podStartE2EDuration="1.102312218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:04:25.100722097 +0000 UTC m=+2311.645125893" watchObservedRunningTime="2026-07-14 18:04:25.102312218 +0000 UTC m=+2311.646716013"
Jul 14 18:04:29.038484 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.038266    2567 patch_prober.go:28] interesting pod/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 14 18:04:29.038484 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.038376    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 14 18:04:29.104411 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.104378    2567 generic.go:356] "Generic (PLEG): container finished" podID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerID="ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff" exitCode=0
Jul 14 18:04:29.104557 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.104453    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" event={"ID":"cd49a5a8-05df-4c25-856a-24f65dc4d491","Type":"ContainerDied","Data":"ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"}
Jul 14 18:04:29.104836 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.104822    2567 scope.go:117] "RemoveContainer" containerID="ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"
Jul 14 18:04:29.105883 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.105849    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:04:29.106002 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.105920    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:04:29.106002 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:29.105951    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:04:30.110554 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:30.110521    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" event={"ID":"cd49a5a8-05df-4c25-856a-24f65dc4d491","Type":"ContainerStarted","Data":"de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b"}
Jul 14 18:04:30.500494 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:30.500413    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:30.500764 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:30.500744    2567 patch_prober.go:28] interesting pod/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 14 18:04:30.500818 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:30.500799    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 14 18:04:31.500628 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:31.500593    2567 patch_prober.go:28] interesting pod/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 14 18:04:31.501018 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:31.500659    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 14 18:04:32.500852 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:32.500816    2567 patch_prober.go:28] interesting pod/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 14 18:04:32.501246 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:32.500881    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 14 18:04:33.500144 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:33.500107    2567 patch_prober.go:28] interesting pod/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 14 18:04:33.500329 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:33.500168    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 14 18:04:34.504197 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:34.504167    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:34.504685 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:34.504563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:34.505292 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:34.505272    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:04:34.797647 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:34.797573    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:34.797780 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:34.797609    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:34.810047 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:34.810029    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:04:35.142287 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:04:35.142260    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:05:08.298752 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:08.298681    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"]
Jul 14 18:05:23.317689 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.317648    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth" podUID="99caf5a7-431a-4d4d-b770-cf110f2e7212" containerName="main" containerID="cri-o://775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b" gracePeriod=45
Jul 14 18:05:23.561932 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.561908    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:05:23.601211 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601190    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-model-cache\") pod \"99caf5a7-431a-4d4d-b770-cf110f2e7212\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") "
Jul 14 18:05:23.601378 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601248    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfhwn\" (UniqueName: \"kubernetes.io/projected/99caf5a7-431a-4d4d-b770-cf110f2e7212-kube-api-access-xfhwn\") pod \"99caf5a7-431a-4d4d-b770-cf110f2e7212\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") "
Jul 14 18:05:23.601378 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601295    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-home\") pod \"99caf5a7-431a-4d4d-b770-cf110f2e7212\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") "
Jul 14 18:05:23.601378 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601327    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caf5a7-431a-4d4d-b770-cf110f2e7212-tls-certs\") pod \"99caf5a7-431a-4d4d-b770-cf110f2e7212\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") "
Jul 14 18:05:23.601558 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601395    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-dshm\") pod \"99caf5a7-431a-4d4d-b770-cf110f2e7212\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") "
Jul 14 18:05:23.601558 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601434    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-model-cache" (OuterVolumeSpecName: "model-cache") pod "99caf5a7-431a-4d4d-b770-cf110f2e7212" (UID: "99caf5a7-431a-4d4d-b770-cf110f2e7212"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:05:23.601558 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601442    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-tmp-dir\") pod \"99caf5a7-431a-4d4d-b770-cf110f2e7212\" (UID: \"99caf5a7-431a-4d4d-b770-cf110f2e7212\") "
Jul 14 18:05:23.601558 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601529    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-home" (OuterVolumeSpecName: "home") pod "99caf5a7-431a-4d4d-b770-cf110f2e7212" (UID: "99caf5a7-431a-4d4d-b770-cf110f2e7212"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:05:23.601755 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601712    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99caf5a7-431a-4d4d-b770-cf110f2e7212" (UID: "99caf5a7-431a-4d4d-b770-cf110f2e7212"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:05:23.601806 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601780    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-model-cache\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:05:23.601806 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.601799    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-home\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:05:23.603440 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.603409    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99caf5a7-431a-4d4d-b770-cf110f2e7212-kube-api-access-xfhwn" (OuterVolumeSpecName: "kube-api-access-xfhwn") pod "99caf5a7-431a-4d4d-b770-cf110f2e7212" (UID: "99caf5a7-431a-4d4d-b770-cf110f2e7212"). InnerVolumeSpecName "kube-api-access-xfhwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:05:23.603540 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.603458    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-dshm" (OuterVolumeSpecName: "dshm") pod "99caf5a7-431a-4d4d-b770-cf110f2e7212" (UID: "99caf5a7-431a-4d4d-b770-cf110f2e7212"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:05:23.603587 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.603572    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99caf5a7-431a-4d4d-b770-cf110f2e7212-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99caf5a7-431a-4d4d-b770-cf110f2e7212" (UID: "99caf5a7-431a-4d4d-b770-cf110f2e7212"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:05:23.702382 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.702361    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-dshm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:05:23.702382 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.702380    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99caf5a7-431a-4d4d-b770-cf110f2e7212-tmp-dir\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:05:23.702513 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.702389    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhwn\" (UniqueName: \"kubernetes.io/projected/99caf5a7-431a-4d4d-b770-cf110f2e7212-kube-api-access-xfhwn\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:05:23.702513 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:23.702399    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99caf5a7-431a-4d4d-b770-cf110f2e7212-tls-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:05:24.339252 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.339224    2567 generic.go:356] "Generic (PLEG): container finished" podID="99caf5a7-431a-4d4d-b770-cf110f2e7212" containerID="775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b" exitCode=0
Jul 14 18:05:24.339689 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.339264    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"
Jul 14 18:05:24.339689 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.339267    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth" event={"ID":"99caf5a7-431a-4d4d-b770-cf110f2e7212","Type":"ContainerDied","Data":"775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b"}
Jul 14 18:05:24.339689 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.339316    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth" event={"ID":"99caf5a7-431a-4d4d-b770-cf110f2e7212","Type":"ContainerDied","Data":"c1232922ea34358c8906837733e6544f61b3fdd2c0828d8c9554096853e1b5aa"}
Jul 14 18:05:24.339689 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.339362    2567 scope.go:117] "RemoveContainer" containerID="775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b"
Jul 14 18:05:24.347518 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.347502    2567 scope.go:117] "RemoveContainer" containerID="775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b"
Jul 14 18:05:24.347748 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:05:24.347732    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b\": container with ID starting with 775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b not found: ID does not exist" containerID="775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b"
Jul 14 18:05:24.347798 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.347755    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b"} err="failed to get container status \"775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b\": rpc error: code = NotFound desc = could not find container \"775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b\": container with ID starting with 775d56c9c79a3cd11c354edb3ed0e194401a489809e940e427e9c133346e742b not found: ID does not exist"
Jul 14 18:05:24.357433 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.357410    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"]
Jul 14 18:05:24.361580 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:24.361562    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4ef174/test-model-4ef174-kserve-d95898f9f-f6zth"]
Jul 14 18:05:26.051495 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:26.051463    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99caf5a7-431a-4d4d-b770-cf110f2e7212" path="/var/lib/kubelet/pods/99caf5a7-431a-4d4d-b770-cf110f2e7212/volumes"
Jul 14 18:05:53.990925 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:53.990894    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 18:05:54.000362 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:05:54.000328    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 18:06:01.275980 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:01.275941    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"]
Jul 14 18:06:01.276488 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:01.276156    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" containerID="cri-o://de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b" gracePeriod=30
Jul 14 18:06:04.507032 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:04.506983    2567 patch_prober.go:28] interesting pod/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 14 18:06:04.507521 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:04.507067    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 14 18:06:06.526147 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.526120    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:06:06.526590 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.526566    2567 generic.go:356] "Generic (PLEG): container finished" podID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerID="de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b" exitCode=0
Jul 14 18:06:06.526673 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.526645    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" event={"ID":"cd49a5a8-05df-4c25-856a-24f65dc4d491","Type":"ContainerDied","Data":"de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b"}
Jul 14 18:06:06.526728 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.526703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d" event={"ID":"cd49a5a8-05df-4c25-856a-24f65dc4d491","Type":"ContainerDied","Data":"b7f94d3c2ecc903761e5ea9f923ff32ac2321bb618f90835f3b6293cf5847b50"}
Jul 14 18:06:06.526778 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.526727    2567 scope.go:117] "RemoveContainer" containerID="de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b"
Jul 14 18:06:06.535751 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.535731    2567 scope.go:117] "RemoveContainer" containerID="ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"
Jul 14 18:06:06.543657 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.543637    2567 scope.go:117] "RemoveContainer" containerID="de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b"
Jul 14 18:06:06.543934 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:06:06.543914    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b\": container with ID starting with de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b not found: ID does not exist" containerID="de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b"
Jul 14 18:06:06.543978 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.543944    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b"} err="failed to get container status \"de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b\": rpc error: code = NotFound desc = could not find container \"de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b\": container with ID starting with de32a2c65ba946355785c18d4245329948bd0285c33d4257fe120b3c39cdb80b not found: ID does not exist"
Jul 14 18:06:06.543978 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.543963    2567 scope.go:117] "RemoveContainer" containerID="ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"
Jul 14 18:06:06.544217 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:06:06.544200    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff\": container with ID starting with ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff not found: ID does not exist" containerID="ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"
Jul 14 18:06:06.544272 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.544219    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff"} err="failed to get container status \"ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff\": rpc error: code = NotFound desc = could not find container \"ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff\": container with ID starting with ff28166067f617531f72ca442f618a851bc1aafc943ac59378b4e20bf11ffaff not found: ID does not exist"
Jul 14 18:06:06.571305 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571273    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-podinfo\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571510 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571325    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-certs\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571510 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571414    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-socket\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571510 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571442    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-token\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571510 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571481    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-data\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571510 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571509    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd49a5a8-05df-4c25-856a-24f65dc4d491-istiod-ca-cert\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571767 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571543    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-envoy\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571767 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571569    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-599t9\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-kube-api-access-599t9\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571767 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571615    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-credential-socket\") pod \"cd49a5a8-05df-4c25-856a-24f65dc4d491\" (UID: \"cd49a5a8-05df-4c25-856a-24f65dc4d491\") "
Jul 14 18:06:06.571767 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571682    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:06.571767 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571690    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:06.572008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571945    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.572008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571967    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-workload-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.572008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.571996    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd49a5a8-05df-4c25-856a-24f65dc4d491-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:06:06.572217 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.572178    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:06.572623 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.572560    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-data" (OuterVolumeSpecName: "istio-data") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:06.574401 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.574364    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-kube-api-access-599t9" (OuterVolumeSpecName: "kube-api-access-599t9") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "kube-api-access-599t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:06:06.574568 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.574544    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 18:06:06.574568 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.574555    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:06:06.574700 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.574623    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-token" (OuterVolumeSpecName: "istio-token") pod "cd49a5a8-05df-4c25-856a-24f65dc4d491" (UID: "cd49a5a8-05df-4c25-856a-24f65dc4d491"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:06:06.673185 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673149    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-credential-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.673185 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673181    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-podinfo\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.673185 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673191    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-token\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.673614 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673200    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-data\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.673614 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673208    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd49a5a8-05df-4c25-856a-24f65dc4d491-istiod-ca-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.673614 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673216    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd49a5a8-05df-4c25-856a-24f65dc4d491-istio-envoy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:06.673614 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:06.673224    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599t9\" (UniqueName: \"kubernetes.io/projected/cd49a5a8-05df-4c25-856a-24f65dc4d491-kube-api-access-599t9\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:06:07.532070 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:07.532043    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"
Jul 14 18:06:07.561631 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:07.561605    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"]
Jul 14 18:06:07.563912 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:07.563886    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2d5136-openshift-default-6c9d8bfd97-pq87d"]
Jul 14 18:06:08.050892 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:08.050860    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" path="/var/lib/kubelet/pods/cd49a5a8-05df-4c25-856a-24f65dc4d491/volumes"
Jul 14 18:06:10.905500 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905465    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"]
Jul 14 18:06:10.905855 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905824    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99caf5a7-431a-4d4d-b770-cf110f2e7212" containerName="main"
Jul 14 18:06:10.905855 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905835    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="99caf5a7-431a-4d4d-b770-cf110f2e7212" containerName="main"
Jul 14 18:06:10.905927 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905865    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy"
Jul 14 18:06:10.905927 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905870    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy"
Jul 14 18:06:10.905927 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905877    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy"
Jul 14 18:06:10.905927 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905882    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy"
Jul 14 18:06:10.906049 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905930    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy"
Jul 14 18:06:10.906049 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.905940    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="99caf5a7-431a-4d4d-b770-cf110f2e7212" containerName="main"
Jul 14 18:06:10.910448 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.910418    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:10.913687 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.913664    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e9d9a7-openshift-default-dockercfg-zj9c7\""
Jul 14 18:06:10.927187 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:10.927163    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"]
Jul 14 18:06:11.010786 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.010752    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-istio-token\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.010962 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.010789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-credential-socket\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.010962 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.010812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77mw7\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-kube-api-access-77mw7\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.010962 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.010935    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-envoy\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.011113 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.010973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef04090-414c-4c9a-b95c-0a7368654db7-istio-podinfo\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.011113 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.011072    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-socket\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.011113 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.011099    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-certs\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.011271 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.011116    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-data\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.011271 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.011162    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef04090-414c-4c9a-b95c-0a7368654db7-istiod-ca-cert\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112444 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112411    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-istio-token\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112600 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112459    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-credential-socket\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112600 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112488    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77mw7\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-kube-api-access-77mw7\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112721 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-envoy\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112721 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112672    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef04090-414c-4c9a-b95c-0a7368654db7-istio-podinfo\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112828 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-socket\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112828 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112753    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-certs\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112828 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-data\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.112994 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.112832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef04090-414c-4c9a-b95c-0a7368654db7-istiod-ca-cert\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.113157 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.113129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-certs\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.113242 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.113165    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-socket\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.113242 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.113234    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-credential-socket\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.113331 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.113242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-data\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.113680 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.113657    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef04090-414c-4c9a-b95c-0a7368654db7-istiod-ca-cert\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.115026 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.114994    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-envoy\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.115199 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.115183    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef04090-414c-4c9a-b95c-0a7368654db7-istio-podinfo\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.124732 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.124712    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-istio-token\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.126385 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.126365    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77mw7\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-kube-api-access-77mw7\") pod \"e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") " pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.222916 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.222831    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:11.359561 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.359537    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"]
Jul 14 18:06:11.362106 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:06:11.362077    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef04090_414c_4c9a_b95c_0a7368654db7.slice/crio-e87906038fdc91be8fb5cf447d73b2a84f854b2eb68933fccfa1cf474e800033 WatchSource:0}: Error finding container e87906038fdc91be8fb5cf447d73b2a84f854b2eb68933fccfa1cf474e800033: Status 404 returned error can't find the container with id e87906038fdc91be8fb5cf447d73b2a84f854b2eb68933fccfa1cf474e800033
Jul 14 18:06:11.364121 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.364089    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:06:11.364195 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.364156    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:06:11.364195 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.364185    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:06:11.551460 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.551365    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" event={"ID":"eef04090-414c-4c9a-b95c-0a7368654db7","Type":"ContainerStarted","Data":"c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"}
Jul 14 18:06:11.551460 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.551430    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" event={"ID":"eef04090-414c-4c9a-b95c-0a7368654db7","Type":"ContainerStarted","Data":"e87906038fdc91be8fb5cf447d73b2a84f854b2eb68933fccfa1cf474e800033"}
Jul 14 18:06:11.581290 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:11.581242    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" podStartSLOduration=1.5812260249999999 podStartE2EDuration="1.581226025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:06:11.579970972 +0000 UTC m=+2418.124374767" watchObservedRunningTime="2026-07-14 18:06:11.581226025 +0000 UTC m=+2418.125629890"
Jul 14 18:06:12.223594 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:12.223563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:12.228605 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:12.228582    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:12.556415 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:12.556318    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:12.557451 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:12.557430    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:25.612089 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:25.612054    2567 generic.go:356] "Generic (PLEG): container finished" podID="eef04090-414c-4c9a-b95c-0a7368654db7" containerID="c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076" exitCode=0
Jul 14 18:06:25.612508 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:25.612125    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" event={"ID":"eef04090-414c-4c9a-b95c-0a7368654db7","Type":"ContainerDied","Data":"c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"}
Jul 14 18:06:25.612569 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:25.612554    2567 scope.go:117] "RemoveContainer" containerID="c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"
Jul 14 18:06:25.613737 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:25.613706    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:06:25.613819 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:25.613790    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:06:25.613862 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:25.613849    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:06:26.618119 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:26.618077    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" event={"ID":"eef04090-414c-4c9a-b95c-0a7368654db7","Type":"ContainerStarted","Data":"0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd"}
Jul 14 18:06:27.223504 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:27.223464    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:27.224635 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:27.224610    2567 patch_prober.go:28] interesting pod/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 14 18:06:27.224773 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:27.224658    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 14 18:06:27.557092 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:27.556988    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:28.224034 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:28.223993    2567 patch_prober.go:28] interesting pod/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 14 18:06:28.224457 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:28.224053    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 14 18:06:29.224109 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:29.224070    2567 patch_prober.go:28] interesting pod/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 14 18:06:29.224536 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:29.224127    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 14 18:06:30.223529 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:30.223492    2567 patch_prober.go:28] interesting pod/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Jul 14 18:06:30.223746 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:30.223556    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Jul 14 18:06:31.228162 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:31.228122    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:06:31.229240 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:06:31.229216    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:07:17.925009 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:17.924969    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"]
Jul 14 18:07:17.925583 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:17.925295    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy" containerID="cri-o://0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd" gracePeriod=30
Jul 14 18:07:23.178066 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.178033    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:07:23.285676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285576    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-certs\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.285676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285612    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-data\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.285676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285636    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef04090-414c-4c9a-b95c-0a7368654db7-istiod-ca-cert\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.285676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285672    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-credential-socket\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.286025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285708    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-socket\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.286025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285781    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-istio-token\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.286025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77mw7\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-kube-api-access-77mw7\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.286025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285831    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef04090-414c-4c9a-b95c-0a7368654db7-istio-podinfo\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.286025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285859    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-envoy\") pod \"eef04090-414c-4c9a-b95c-0a7368654db7\" (UID: \"eef04090-414c-4c9a-b95c-0a7368654db7\") "
Jul 14 18:07:23.286025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.285934    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:07:23.286337 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286025    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:07:23.286337 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286131    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef04090-414c-4c9a-b95c-0a7368654db7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:07:23.286474 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286440    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.286474 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286461    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eef04090-414c-4c9a-b95c-0a7368654db7-istiod-ca-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.286564 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286476    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-credential-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.286564 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286494    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:07:23.286707 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.286677    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-data" (OuterVolumeSpecName: "istio-data") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:07:23.288206 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.288179    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eef04090-414c-4c9a-b95c-0a7368654db7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 18:07:23.288206 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.288189    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-istio-token" (OuterVolumeSpecName: "istio-token") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:07:23.288480 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.288441    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-kube-api-access-77mw7" (OuterVolumeSpecName: "kube-api-access-77mw7") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "kube-api-access-77mw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:07:23.288545 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.288525    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eef04090-414c-4c9a-b95c-0a7368654db7" (UID: "eef04090-414c-4c9a-b95c-0a7368654db7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:07:23.387319 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.387276    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-workload-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.387319 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.387312    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-istio-token\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.387319 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.387321    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77mw7\" (UniqueName: \"kubernetes.io/projected/eef04090-414c-4c9a-b95c-0a7368654db7-kube-api-access-77mw7\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.387319 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.387330    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eef04090-414c-4c9a-b95c-0a7368654db7-istio-podinfo\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.387618 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.387339    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-envoy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.387618 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.387379    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eef04090-414c-4c9a-b95c-0a7368654db7-istio-data\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:07:23.882215 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.882177    2567 generic.go:356] "Generic (PLEG): container finished" podID="eef04090-414c-4c9a-b95c-0a7368654db7" containerID="0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd" exitCode=0
Jul 14 18:07:23.882400 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.882236    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"
Jul 14 18:07:23.882400 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.882259    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" event={"ID":"eef04090-414c-4c9a-b95c-0a7368654db7","Type":"ContainerDied","Data":"0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd"}
Jul 14 18:07:23.882400 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.882315    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn" event={"ID":"eef04090-414c-4c9a-b95c-0a7368654db7","Type":"ContainerDied","Data":"e87906038fdc91be8fb5cf447d73b2a84f854b2eb68933fccfa1cf474e800033"}
Jul 14 18:07:23.882400 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.882337    2567 scope.go:117] "RemoveContainer" containerID="0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd"
Jul 14 18:07:23.892263 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.892246    2567 scope.go:117] "RemoveContainer" containerID="c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"
Jul 14 18:07:23.900907 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.900880    2567 scope.go:117] "RemoveContainer" containerID="0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd"
Jul 14 18:07:23.901193 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:07:23.901175    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd\": container with ID starting with 0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd not found: ID does not exist" containerID="0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd"
Jul 14 18:07:23.901266 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.901200    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd"} err="failed to get container status \"0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd\": rpc error: code = NotFound desc = could not find container \"0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd\": container with ID starting with 0d9cff6db20fe354b7e59a6f7e0f57c40dc7e282b11a0c9442cb7da16aa893cd not found: ID does not exist"
Jul 14 18:07:23.901266 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.901219    2567 scope.go:117] "RemoveContainer" containerID="c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"
Jul 14 18:07:23.901486 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:07:23.901466    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076\": container with ID starting with c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076 not found: ID does not exist" containerID="c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"
Jul 14 18:07:23.901540 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.901490    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076"} err="failed to get container status \"c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076\": rpc error: code = NotFound desc = could not find container \"c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076\": container with ID starting with c2e8dc6e84af1ac5e883bce7bf7ab31822bb1b7bfde20f33d46bf77c1109a076 not found: ID does not exist"
Jul 14 18:07:23.907149 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.907126    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"]
Jul 14 18:07:23.911728 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:23.911708    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e9d9a7-openshift-default-7b6d564455-bbwgn"]
Jul 14 18:07:24.051657 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:24.051622    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" path="/var/lib/kubelet/pods/eef04090-414c-4c9a-b95c-0a7368654db7/volumes"
Jul 14 18:07:27.174221 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.174185    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"]
Jul 14 18:07:27.174635 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.174550    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy"
Jul 14 18:07:27.174635 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.174563    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy"
Jul 14 18:07:27.174635 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.174623    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy"
Jul 14 18:07:27.174635 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.174635    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd49a5a8-05df-4c25-856a-24f65dc4d491" containerName="istio-proxy"
Jul 14 18:07:27.178948 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.178925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.186313 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.186290    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4f6333-openshift-default-dockercfg-rqj7w\""
Jul 14 18:07:27.201149 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.201113    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"]
Jul 14 18:07:27.324484 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-socket\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324504    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-credential-socket\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istiod-ca-cert\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324618    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-envoy\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324676 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324641    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-podinfo\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324866 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-data\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324866 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324762    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-certs\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324866 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-token\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.324866 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.324813    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976lc\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-kube-api-access-976lc\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.425817 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.425730    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-certs\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.425817 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.425767    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-token\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.425817 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.425792    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976lc\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-kube-api-access-976lc\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426074 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.425836    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-socket\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426074 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.425865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-credential-socket\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426074 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.425891    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istiod-ca-cert\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426205 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426150    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-certs\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426245 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426215    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-socket\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426283 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426245    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-credential-socket\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426317 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426275    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-envoy\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426382 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426318    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-podinfo\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426435 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426388    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-data\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426593 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426571    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istiod-ca-cert\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.426715 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.426694    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-data\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.428229 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.428202    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-envoy\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.428685 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.428665    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-podinfo\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.438408 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.438384    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-976lc\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-kube-api-access-976lc\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.438820 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.438799    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-token\") pod \"e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") " pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.489752 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.489719    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:27.625280 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.625256    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"]
Jul 14 18:07:27.627328 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:07:27.627301    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6b21a6_0e5e_4109_8612_41d7a420f9c6.slice/crio-b15e1acee0aa15e231b9c3823e018332501806a2220575e4efd70a5dea4fea1f WatchSource:0}: Error finding container b15e1acee0aa15e231b9c3823e018332501806a2220575e4efd70a5dea4fea1f: Status 404 returned error can't find the container with id b15e1acee0aa15e231b9c3823e018332501806a2220575e4efd70a5dea4fea1f
Jul 14 18:07:27.629590 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.629558    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:07:27.629679 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.629624    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:07:27.629679 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.629653    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:07:27.903547 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.903513    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" event={"ID":"5d6b21a6-0e5e-4109-8612-41d7a420f9c6","Type":"ContainerStarted","Data":"22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"}
Jul 14 18:07:27.903741 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:27.903568    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" event={"ID":"5d6b21a6-0e5e-4109-8612-41d7a420f9c6","Type":"ContainerStarted","Data":"b15e1acee0aa15e231b9c3823e018332501806a2220575e4efd70a5dea4fea1f"}
Jul 14 18:07:28.489898 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:28.489853    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:28.494674 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:28.494645    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:28.520366 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:28.520297    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" podStartSLOduration=1.520279422 podStartE2EDuration="1.520279422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:07:27.924971045 +0000 UTC m=+2494.469374840" watchObservedRunningTime="2026-07-14 18:07:28.520279422 +0000 UTC m=+2495.064683216"
Jul 14 18:07:28.908447 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:28.908418    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:28.909443 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:28.909420    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:41.977586 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:41.977559    2567 generic.go:356] "Generic (PLEG): container finished" podID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerID="22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4" exitCode=0
Jul 14 18:07:41.977948 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:41.977627    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" event={"ID":"5d6b21a6-0e5e-4109-8612-41d7a420f9c6","Type":"ContainerDied","Data":"22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"}
Jul 14 18:07:41.977992 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:41.977951    2567 scope.go:117] "RemoveContainer" containerID="22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"
Jul 14 18:07:41.978851 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:41.978822    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:07:41.978956 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:41.978892    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:07:41.978956 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:41.978926    2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 14 18:07:42.986473 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:42.986441    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" event={"ID":"5d6b21a6-0e5e-4109-8612-41d7a420f9c6","Type":"ContainerStarted","Data":"727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139"}
Jul 14 18:07:43.490217 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:43.490185    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:43.491032 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:43.491008    2567 patch_prober.go:28] interesting pod/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 14 18:07:43.491123 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:43.491055    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 14 18:07:43.908479 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:43.908449    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:44.490987 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:44.490951    2567 patch_prober.go:28] interesting pod/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 14 18:07:44.491456 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:44.491014    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 14 18:07:45.490091 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:45.490051    2567 patch_prober.go:28] interesting pod/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 14 18:07:45.490297 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:45.490109    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 14 18:07:46.490531 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:46.490493    2567 patch_prober.go:28] interesting pod/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Jul 14 18:07:46.490936 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:46.490559    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Jul 14 18:07:47.494531 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:47.494499    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:47.495646 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:47.495626    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:07:58.893037 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.892998    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"]
Jul 14 18:07:58.893434 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.893374    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy"
Jul 14 18:07:58.893434 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.893390    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy"
Jul 14 18:07:58.893511 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.893473    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef04090-414c-4c9a-b95c-0a7368654db7" containerName="istio-proxy"
Jul 14 18:07:58.897731 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.897710    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:58.900403 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.900380    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8bfa74\"/\"kube-root-ca.crt\""
Jul 14 18:07:58.901788 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.901765    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bfa74\"/\"default-dockercfg-56rwl\""
Jul 14 18:07:58.901890 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.901785    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8bfa74\"/\"openshift-service-ca.crt\""
Jul 14 18:07:58.901890 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.901774    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bfa74\"/\"auth-test-model-8bfa74-kserve-self-signed-certs\""
Jul 14 18:07:58.906155 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.906134    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"]
Jul 14 18:07:58.996250 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.996222    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cdw\" (UniqueName: \"kubernetes.io/projected/5ebf9424-df09-4974-999c-aa7d9d7b11d2-kube-api-access-k8cdw\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:58.996380 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.996256    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tmp-dir\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:58.996380 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.996336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-model-cache\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:58.996380 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.996375    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tls-certs\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:58.996511 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.996402    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-home\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:58.996511 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:58.996431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-dshm\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097456 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097428    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-model-cache\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097581 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097468    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tls-certs\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097633 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097582    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-home\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097633 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-dshm\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097720 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097652    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cdw\" (UniqueName: \"kubernetes.io/projected/5ebf9424-df09-4974-999c-aa7d9d7b11d2-kube-api-access-k8cdw\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097720 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tmp-dir\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097850 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097829    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-model-cache\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.097912 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.097890    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-home\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.098087 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.098060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tmp-dir\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.099748 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.099720    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-dshm\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.099890 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.099874    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tls-certs\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.106174 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.106153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cdw\" (UniqueName: \"kubernetes.io/projected/5ebf9424-df09-4974-999c-aa7d9d7b11d2-kube-api-access-k8cdw\") pod \"auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") " pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.207328 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.207268    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:07:59.340314 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:07:59.340292    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"]
Jul 14 18:07:59.341897 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:07:59.341859    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ebf9424_df09_4974_999c_aa7d9d7b11d2.slice/crio-69e9863efd87030cf5cd4928ad485b03e0759a3c95fc1f955f8b25a53d218113 WatchSource:0}: Error finding container 69e9863efd87030cf5cd4928ad485b03e0759a3c95fc1f955f8b25a53d218113: Status 404 returned error can't find the container with id 69e9863efd87030cf5cd4928ad485b03e0759a3c95fc1f955f8b25a53d218113
Jul 14 18:08:00.064792 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:00.064760    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc" event={"ID":"5ebf9424-df09-4974-999c-aa7d9d7b11d2","Type":"ContainerStarted","Data":"d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d"}
Jul 14 18:08:00.065206 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:00.064798    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc" event={"ID":"5ebf9424-df09-4974-999c-aa7d9d7b11d2","Type":"ContainerStarted","Data":"69e9863efd87030cf5cd4928ad485b03e0759a3c95fc1f955f8b25a53d218113"}
Jul 14 18:08:00.084315 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:00.084272    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc" podStartSLOduration=2.084257506 podStartE2EDuration="2.084257506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:08:00.083370696 +0000 UTC m=+2526.627774493" watchObservedRunningTime="2026-07-14 18:08:00.084257506 +0000 UTC m=+2526.628661300"
Jul 14 18:08:09.207842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:09.207807    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:08:09.207842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:09.207853    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:08:09.220188 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:09.220167    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:08:10.116527 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:08:10.116496    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:10:09.337567 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:09.337527    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"]
Jul 14 18:10:21.992440 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:21.992410    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"]
Jul 14 18:10:21.995915 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:21.995895    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:21.998371 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:21.998338    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8bfa74\"/\"sub-test-model-8bfa74-kserve-self-signed-certs\""
Jul 14 18:10:22.006535 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.006514    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"]
Jul 14 18:10:22.067434 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.067410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp7lx\" (UniqueName: \"kubernetes.io/projected/8308306c-2024-4db1-a9a4-d7403ad03747-kube-api-access-fp7lx\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.067544 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.067461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8308306c-2024-4db1-a9a4-d7403ad03747-tls-certs\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.067544 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.067492    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-model-cache\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.067544 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.067519    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-home\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.067544 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.067535    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-tmp-dir\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.067697 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.067628    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-dshm\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.168360 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.168320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-dshm\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.168473 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.168419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp7lx\" (UniqueName: \"kubernetes.io/projected/8308306c-2024-4db1-a9a4-d7403ad03747-kube-api-access-fp7lx\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.168758 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.168512    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8308306c-2024-4db1-a9a4-d7403ad03747-tls-certs\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.168842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.168802    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-model-cache\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.168890 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.168841    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-home\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.168890 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.168865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-tmp-dir\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.169198 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.169174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-model-cache\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.169281 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.169199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-home\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.169281 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.169247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-tmp-dir\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.170758 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.170732    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-dshm\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.170920 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.170903    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8308306c-2024-4db1-a9a4-d7403ad03747-tls-certs\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.176526 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.176505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp7lx\" (UniqueName: \"kubernetes.io/projected/8308306c-2024-4db1-a9a4-d7403ad03747-kube-api-access-fp7lx\") pod \"sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") " pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.304568 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.304482    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:22.430328 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.430303    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"]
Jul 14 18:10:22.432671 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:10:22.432643    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8308306c_2024_4db1_a9a4_d7403ad03747.slice/crio-1e0d8863fe6079503620a86f8d8c6ee43db7c9cf85e75afb429acfa075962fe7 WatchSource:0}: Error finding container 1e0d8863fe6079503620a86f8d8c6ee43db7c9cf85e75afb429acfa075962fe7: Status 404 returned error can't find the container with id 1e0d8863fe6079503620a86f8d8c6ee43db7c9cf85e75afb429acfa075962fe7
Jul 14 18:10:22.697779 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.697739    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm" event={"ID":"8308306c-2024-4db1-a9a4-d7403ad03747","Type":"ContainerStarted","Data":"66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de"}
Jul 14 18:10:22.697958 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.697797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm" event={"ID":"8308306c-2024-4db1-a9a4-d7403ad03747","Type":"ContainerStarted","Data":"1e0d8863fe6079503620a86f8d8c6ee43db7c9cf85e75afb429acfa075962fe7"}
Jul 14 18:10:22.725755 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:22.725709    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm" podStartSLOduration=1.7256938480000001 podStartE2EDuration="1.725693848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:22.724461306 +0000 UTC m=+2669.268865100" watchObservedRunningTime="2026-07-14 18:10:22.725693848 +0000 UTC m=+2669.270097643"
Jul 14 18:10:24.357633 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.357594    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc" podUID="5ebf9424-df09-4974-999c-aa7d9d7b11d2" containerName="main" containerID="cri-o://d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d" gracePeriod=45
Jul 14 18:10:24.604448 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.604422    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:10:24.693424 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693328    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8cdw\" (UniqueName: \"kubernetes.io/projected/5ebf9424-df09-4974-999c-aa7d9d7b11d2-kube-api-access-k8cdw\") pod \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") "
Jul 14 18:10:24.693424 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693391    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-home\") pod \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") "
Jul 14 18:10:24.693602 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693426    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tmp-dir\") pod \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") "
Jul 14 18:10:24.693602 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693444    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tls-certs\") pod \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") "
Jul 14 18:10:24.693602 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693463    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-model-cache\") pod \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") "
Jul 14 18:10:24.693602 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693524    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-dshm\") pod \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\" (UID: \"5ebf9424-df09-4974-999c-aa7d9d7b11d2\") "
Jul 14 18:10:24.693818 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693689    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ebf9424-df09-4974-999c-aa7d9d7b11d2" (UID: "5ebf9424-df09-4974-999c-aa7d9d7b11d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:24.693818 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693714    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-home" (OuterVolumeSpecName: "home") pod "5ebf9424-df09-4974-999c-aa7d9d7b11d2" (UID: "5ebf9424-df09-4974-999c-aa7d9d7b11d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:24.693818 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693777    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ebf9424-df09-4974-999c-aa7d9d7b11d2" (UID: "5ebf9424-df09-4974-999c-aa7d9d7b11d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:24.693934 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693899    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-home\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:10:24.693934 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693919    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tmp-dir\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:10:24.694006 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.693933    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-model-cache\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:10:24.695561 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.695538    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-dshm" (OuterVolumeSpecName: "dshm") pod "5ebf9424-df09-4974-999c-aa7d9d7b11d2" (UID: "5ebf9424-df09-4974-999c-aa7d9d7b11d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:10:24.695683 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.695564    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebf9424-df09-4974-999c-aa7d9d7b11d2-kube-api-access-k8cdw" (OuterVolumeSpecName: "kube-api-access-k8cdw") pod "5ebf9424-df09-4974-999c-aa7d9d7b11d2" (UID: "5ebf9424-df09-4974-999c-aa7d9d7b11d2"). InnerVolumeSpecName "kube-api-access-k8cdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:10:24.695683 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.695592    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ebf9424-df09-4974-999c-aa7d9d7b11d2" (UID: "5ebf9424-df09-4974-999c-aa7d9d7b11d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:10:24.708658 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.708630    2567 generic.go:356] "Generic (PLEG): container finished" podID="5ebf9424-df09-4974-999c-aa7d9d7b11d2" containerID="d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d" exitCode=0
Jul 14 18:10:24.708747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.708669    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc" event={"ID":"5ebf9424-df09-4974-999c-aa7d9d7b11d2","Type":"ContainerDied","Data":"d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d"}
Jul 14 18:10:24.708747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.708671    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"
Jul 14 18:10:24.708747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.708710    2567 scope.go:117] "RemoveContainer" containerID="d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d"
Jul 14 18:10:24.708868 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.708697    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc" event={"ID":"5ebf9424-df09-4974-999c-aa7d9d7b11d2","Type":"ContainerDied","Data":"69e9863efd87030cf5cd4928ad485b03e0759a3c95fc1f955f8b25a53d218113"}
Jul 14 18:10:24.720811 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.717244    2567 scope.go:117] "RemoveContainer" containerID="d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d"
Jul 14 18:10:24.723060 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:10:24.723034    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d\": container with ID starting with d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d not found: ID does not exist" containerID="d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d"
Jul 14 18:10:24.723151 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.723072    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d"} err="failed to get container status \"d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d\": rpc error: code = NotFound desc = could not find container \"d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d\": container with ID starting with d3db166a1761594e285a0636c50c85e23b207935929560f1bd152e722c25da9d not found: ID does not exist"
Jul 14 18:10:24.731432 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.731408    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"]
Jul 14 18:10:24.734886 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.734865    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/auth-test-model-8bfa74-kserve-85bbbf9cf9-hqnzc"]
Jul 14 18:10:24.795282 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.795253    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ebf9424-df09-4974-999c-aa7d9d7b11d2-tls-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:10:24.795282 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.795278    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ebf9424-df09-4974-999c-aa7d9d7b11d2-dshm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:10:24.795443 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:24.795288    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8cdw\" (UniqueName: \"kubernetes.io/projected/5ebf9424-df09-4974-999c-aa7d9d7b11d2-kube-api-access-k8cdw\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:10:26.053326 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:26.053291    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebf9424-df09-4974-999c-aa7d9d7b11d2" path="/var/lib/kubelet/pods/5ebf9424-df09-4974-999c-aa7d9d7b11d2/volumes"
Jul 14 18:10:32.305074 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:32.305031    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:32.305074 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:32.305080    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:32.317988 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:32.317965    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:32.759967 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:32.759936    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:10:43.697831 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.697797    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"]
Jul 14 18:10:43.698274 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.698141    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebf9424-df09-4974-999c-aa7d9d7b11d2" containerName="main"
Jul 14 18:10:43.698274 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.698155    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebf9424-df09-4974-999c-aa7d9d7b11d2" containerName="main"
Jul 14 18:10:43.698274 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.698223    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebf9424-df09-4974-999c-aa7d9d7b11d2" containerName="main"
Jul 14 18:10:43.702356 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.702309    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.709612 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.709565    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"default-dockercfg-nwbq6\""
Jul 14 18:10:43.709759 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.709650    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"sub-test-model-4f6333-kserve-self-signed-certs\""
Jul 14 18:10:43.709759 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.709656    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"kube-root-ca.crt\""
Jul 14 18:10:43.709908 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.709759    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"openshift-service-ca.crt\""
Jul 14 18:10:43.715081 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.715057    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"]
Jul 14 18:10:43.869038 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.869008    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-home\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.869193 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.869080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-dshm\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.869193 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.869127    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-tmp-dir\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.869279 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.869198    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-model-cache\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.869279 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.869237    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680b1201-3799-4d8f-a7c9-050a84c6ed02-tls-certs\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.869279 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.869269    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4wjf\" (UniqueName: \"kubernetes.io/projected/680b1201-3799-4d8f-a7c9-050a84c6ed02-kube-api-access-n4wjf\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.970511 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.970437    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680b1201-3799-4d8f-a7c9-050a84c6ed02-tls-certs\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.970511 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.970488    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4wjf\" (UniqueName: \"kubernetes.io/projected/680b1201-3799-4d8f-a7c9-050a84c6ed02-kube-api-access-n4wjf\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.970712 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.970556    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-home\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.970712 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.970616    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-dshm\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.970712 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.970650    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-tmp-dir\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.970905 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.970703    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-model-cache\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.971042 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.971019    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-home\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.971105 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.971085    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-tmp-dir\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.971147 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.971112    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-model-cache\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.972764 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.972747    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-dshm\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.973013 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.972994    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680b1201-3799-4d8f-a7c9-050a84c6ed02-tls-certs\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:43.979484 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:43.979452    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4wjf\" (UniqueName: \"kubernetes.io/projected/680b1201-3799-4d8f-a7c9-050a84c6ed02-kube-api-access-n4wjf\") pod \"sub-test-model-4f6333-kserve-6d58747dbf-dwfnq\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") " pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:44.013651 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:44.013625    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:44.143387 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:44.143362    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"]
Jul 14 18:10:44.144474 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:10:44.144442    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod680b1201_3799_4d8f_a7c9_050a84c6ed02.slice/crio-acbea444f31e7f5682e88460695d6769c3ec0c112f7eaac17152b9c242f737fc WatchSource:0}: Error finding container acbea444f31e7f5682e88460695d6769c3ec0c112f7eaac17152b9c242f737fc: Status 404 returned error can't find the container with id acbea444f31e7f5682e88460695d6769c3ec0c112f7eaac17152b9c242f737fc
Jul 14 18:10:44.809474 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:44.809445    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq" event={"ID":"680b1201-3799-4d8f-a7c9-050a84c6ed02","Type":"ContainerStarted","Data":"88ed81133ff09598b8f7799e2c21107904f6019843f905f91d5cd9e5e37bf12a"}
Jul 14 18:10:44.809828 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:44.809484    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq" event={"ID":"680b1201-3799-4d8f-a7c9-050a84c6ed02","Type":"ContainerStarted","Data":"acbea444f31e7f5682e88460695d6769c3ec0c112f7eaac17152b9c242f737fc"}
Jul 14 18:10:44.828989 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:44.828941    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq" podStartSLOduration=1.828925186 podStartE2EDuration="1.828925186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:10:44.826511735 +0000 UTC m=+2691.370915523" watchObservedRunningTime="2026-07-14 18:10:44.828925186 +0000 UTC m=+2691.373328981"
Jul 14 18:10:54.013892 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:54.013684    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:54.013892 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:54.013753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:54.018163 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:54.018140    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 18:10:54.028441 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:54.028421    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:10:54.030092 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:54.030075    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-pqtvp_cf8d762e-4ca2-4835-a4c0-e4eba74ad85f/ovn-acl-logging/0.log"
Jul 14 18:10:54.867654 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:10:54.867627    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:11:32.118291 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:32.118261    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"]
Jul 14 18:11:32.912288 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:32.912256    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"]
Jul 14 18:11:47.140307 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.140266    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm" podUID="8308306c-2024-4db1-a9a4-d7403ad03747" containerName="main" containerID="cri-o://66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de" gracePeriod=45
Jul 14 18:11:47.399115 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.399055    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:11:47.421131 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421022    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-dshm\") pod \"8308306c-2024-4db1-a9a4-d7403ad03747\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") "
Jul 14 18:11:47.421131 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421124    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp7lx\" (UniqueName: \"kubernetes.io/projected/8308306c-2024-4db1-a9a4-d7403ad03747-kube-api-access-fp7lx\") pod \"8308306c-2024-4db1-a9a4-d7403ad03747\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") "
Jul 14 18:11:47.421374 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421162    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-tmp-dir\") pod \"8308306c-2024-4db1-a9a4-d7403ad03747\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") "
Jul 14 18:11:47.421374 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421227    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-home\") pod \"8308306c-2024-4db1-a9a4-d7403ad03747\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") "
Jul 14 18:11:47.421374 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421246    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-model-cache\") pod \"8308306c-2024-4db1-a9a4-d7403ad03747\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") "
Jul 14 18:11:47.421374 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421274    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8308306c-2024-4db1-a9a4-d7403ad03747-tls-certs\") pod \"8308306c-2024-4db1-a9a4-d7403ad03747\" (UID: \"8308306c-2024-4db1-a9a4-d7403ad03747\") "
Jul 14 18:11:47.421843 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421597    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-home" (OuterVolumeSpecName: "home") pod "8308306c-2024-4db1-a9a4-d7403ad03747" (UID: "8308306c-2024-4db1-a9a4-d7403ad03747"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:47.421843 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421715    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-home\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:47.422014 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.421869    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-model-cache" (OuterVolumeSpecName: "model-cache") pod "8308306c-2024-4db1-a9a4-d7403ad03747" (UID: "8308306c-2024-4db1-a9a4-d7403ad03747"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:47.422164 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.422125    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8308306c-2024-4db1-a9a4-d7403ad03747" (UID: "8308306c-2024-4db1-a9a4-d7403ad03747"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:47.423503 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.423413    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8308306c-2024-4db1-a9a4-d7403ad03747-kube-api-access-fp7lx" (OuterVolumeSpecName: "kube-api-access-fp7lx") pod "8308306c-2024-4db1-a9a4-d7403ad03747" (UID: "8308306c-2024-4db1-a9a4-d7403ad03747"). InnerVolumeSpecName "kube-api-access-fp7lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:11:47.423922 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.423901    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-dshm" (OuterVolumeSpecName: "dshm") pod "8308306c-2024-4db1-a9a4-d7403ad03747" (UID: "8308306c-2024-4db1-a9a4-d7403ad03747"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:47.424008 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.423919    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8308306c-2024-4db1-a9a4-d7403ad03747-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8308306c-2024-4db1-a9a4-d7403ad03747" (UID: "8308306c-2024-4db1-a9a4-d7403ad03747"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:11:47.522307 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.522271    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-tmp-dir\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:47.522307 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.522304    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-model-cache\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:47.522539 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.522317    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8308306c-2024-4db1-a9a4-d7403ad03747-tls-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:47.522539 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.522328    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8308306c-2024-4db1-a9a4-d7403ad03747-dshm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:47.522539 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.522368    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp7lx\" (UniqueName: \"kubernetes.io/projected/8308306c-2024-4db1-a9a4-d7403ad03747-kube-api-access-fp7lx\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:47.933227 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:47.933180    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq" podUID="680b1201-3799-4d8f-a7c9-050a84c6ed02" containerName="main" containerID="cri-o://88ed81133ff09598b8f7799e2c21107904f6019843f905f91d5cd9e5e37bf12a" gracePeriod=45
Jul 14 18:11:48.105322 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.105292    2567 generic.go:356] "Generic (PLEG): container finished" podID="8308306c-2024-4db1-a9a4-d7403ad03747" containerID="66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de" exitCode=0
Jul 14 18:11:48.105489 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.105369    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"
Jul 14 18:11:48.105489 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.105379    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm" event={"ID":"8308306c-2024-4db1-a9a4-d7403ad03747","Type":"ContainerDied","Data":"66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de"}
Jul 14 18:11:48.105489 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.105425    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm" event={"ID":"8308306c-2024-4db1-a9a4-d7403ad03747","Type":"ContainerDied","Data":"1e0d8863fe6079503620a86f8d8c6ee43db7c9cf85e75afb429acfa075962fe7"}
Jul 14 18:11:48.105489 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.105448    2567 scope.go:117] "RemoveContainer" containerID="66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de"
Jul 14 18:11:48.107559 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.107536    2567 generic.go:356] "Generic (PLEG): container finished" podID="680b1201-3799-4d8f-a7c9-050a84c6ed02" containerID="88ed81133ff09598b8f7799e2c21107904f6019843f905f91d5cd9e5e37bf12a" exitCode=0
Jul 14 18:11:48.107696 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.107625    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq" event={"ID":"680b1201-3799-4d8f-a7c9-050a84c6ed02","Type":"ContainerDied","Data":"88ed81133ff09598b8f7799e2c21107904f6019843f905f91d5cd9e5e37bf12a"}
Jul 14 18:11:48.114460 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.114440    2567 scope.go:117] "RemoveContainer" containerID="66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de"
Jul 14 18:11:48.114813 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:11:48.114787    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de\": container with ID starting with 66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de not found: ID does not exist" containerID="66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de"
Jul 14 18:11:48.114878 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.114829    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de"} err="failed to get container status \"66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de\": rpc error: code = NotFound desc = could not find container \"66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de\": container with ID starting with 66573b68f9ef4e12df23aceb5b23e8b1cf1ff3a50b4a1763d336a70580be70de not found: ID does not exist"
Jul 14 18:11:48.123731 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.123701    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"]
Jul 14 18:11:48.127670 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.127642    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8bfa74/sub-test-model-8bfa74-kserve-77dbb898f9-p6jkm"]
Jul 14 18:11:48.186379 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.186305    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:11:48.228812 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.228768    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680b1201-3799-4d8f-a7c9-050a84c6ed02-tls-certs\") pod \"680b1201-3799-4d8f-a7c9-050a84c6ed02\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") "
Jul 14 18:11:48.228986 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.228879    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-dshm\") pod \"680b1201-3799-4d8f-a7c9-050a84c6ed02\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") "
Jul 14 18:11:48.228986 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.228938    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-tmp-dir\") pod \"680b1201-3799-4d8f-a7c9-050a84c6ed02\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") "
Jul 14 18:11:48.229084 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.229025    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4wjf\" (UniqueName: \"kubernetes.io/projected/680b1201-3799-4d8f-a7c9-050a84c6ed02-kube-api-access-n4wjf\") pod \"680b1201-3799-4d8f-a7c9-050a84c6ed02\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") "
Jul 14 18:11:48.229084 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.229079    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-home\") pod \"680b1201-3799-4d8f-a7c9-050a84c6ed02\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") "
Jul 14 18:11:48.229183 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.229102    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-model-cache\") pod \"680b1201-3799-4d8f-a7c9-050a84c6ed02\" (UID: \"680b1201-3799-4d8f-a7c9-050a84c6ed02\") "
Jul 14 18:11:48.229494 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.229462    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-home" (OuterVolumeSpecName: "home") pod "680b1201-3799-4d8f-a7c9-050a84c6ed02" (UID: "680b1201-3799-4d8f-a7c9-050a84c6ed02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:48.229494 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.229477    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "680b1201-3799-4d8f-a7c9-050a84c6ed02" (UID: "680b1201-3799-4d8f-a7c9-050a84c6ed02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:48.229701 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.229611    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-model-cache" (OuterVolumeSpecName: "model-cache") pod "680b1201-3799-4d8f-a7c9-050a84c6ed02" (UID: "680b1201-3799-4d8f-a7c9-050a84c6ed02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:48.231297 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.231269    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680b1201-3799-4d8f-a7c9-050a84c6ed02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "680b1201-3799-4d8f-a7c9-050a84c6ed02" (UID: "680b1201-3799-4d8f-a7c9-050a84c6ed02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:11:48.231436 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.231295    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680b1201-3799-4d8f-a7c9-050a84c6ed02-kube-api-access-n4wjf" (OuterVolumeSpecName: "kube-api-access-n4wjf") pod "680b1201-3799-4d8f-a7c9-050a84c6ed02" (UID: "680b1201-3799-4d8f-a7c9-050a84c6ed02"). InnerVolumeSpecName "kube-api-access-n4wjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:11:48.231548 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.231530    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-dshm" (OuterVolumeSpecName: "dshm") pod "680b1201-3799-4d8f-a7c9-050a84c6ed02" (UID: "680b1201-3799-4d8f-a7c9-050a84c6ed02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:11:48.330684 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.330646    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-home\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:48.330684 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.330676    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-model-cache\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:48.330684 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.330687    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680b1201-3799-4d8f-a7c9-050a84c6ed02-tls-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:48.330916 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.330697    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-dshm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:48.330916 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.330704    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680b1201-3799-4d8f-a7c9-050a84c6ed02-tmp-dir\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:48.330916 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:48.330712    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4wjf\" (UniqueName: \"kubernetes.io/projected/680b1201-3799-4d8f-a7c9-050a84c6ed02-kube-api-access-n4wjf\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:11:49.114087 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:49.114056    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"
Jul 14 18:11:49.114275 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:49.114055    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq" event={"ID":"680b1201-3799-4d8f-a7c9-050a84c6ed02","Type":"ContainerDied","Data":"acbea444f31e7f5682e88460695d6769c3ec0c112f7eaac17152b9c242f737fc"}
Jul 14 18:11:49.114275 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:49.114180    2567 scope.go:117] "RemoveContainer" containerID="88ed81133ff09598b8f7799e2c21107904f6019843f905f91d5cd9e5e37bf12a"
Jul 14 18:11:49.138672 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:49.138640    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"]
Jul 14 18:11:49.140182 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:49.140161    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/sub-test-model-4f6333-kserve-6d58747dbf-dwfnq"]
Jul 14 18:11:50.051304 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:50.051266    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="680b1201-3799-4d8f-a7c9-050a84c6ed02" path="/var/lib/kubelet/pods/680b1201-3799-4d8f-a7c9-050a84c6ed02/volumes"
Jul 14 18:11:50.051856 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:11:50.051841    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8308306c-2024-4db1-a9a4-d7403ad03747" path="/var/lib/kubelet/pods/8308306c-2024-4db1-a9a4-d7403ad03747/volumes"
Jul 14 18:12:27.289545 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.289515    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"]
Jul 14 18:12:27.289998 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.289866    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="680b1201-3799-4d8f-a7c9-050a84c6ed02" containerName="main"
Jul 14 18:12:27.289998 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.289880    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="680b1201-3799-4d8f-a7c9-050a84c6ed02" containerName="main"
Jul 14 18:12:27.289998 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.289893    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8308306c-2024-4db1-a9a4-d7403ad03747" containerName="main"
Jul 14 18:12:27.289998 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.289898    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8308306c-2024-4db1-a9a4-d7403ad03747" containerName="main"
Jul 14 18:12:27.289998 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.289968    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="680b1201-3799-4d8f-a7c9-050a84c6ed02" containerName="main"
Jul 14 18:12:27.290200 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.290023    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8308306c-2024-4db1-a9a4-d7403ad03747" containerName="main"
Jul 14 18:12:27.293103 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.293082    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.296230 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.296210    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"openshift-service-ca.crt\""
Jul 14 18:12:27.297266 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.297250    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"rate-test-model-4f6333-kserve-self-signed-certs\""
Jul 14 18:12:27.297600 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.297582    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"default-dockercfg-nwbq6\""
Jul 14 18:12:27.297689 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.297654    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4f6333\"/\"kube-root-ca.crt\""
Jul 14 18:12:27.306062 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.306041    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"]
Jul 14 18:12:27.396916 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.396882    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-home\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.397073 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.396922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mvd\" (UniqueName: \"kubernetes.io/projected/038f71e2-e6db-4549-984c-798dda6d667d-kube-api-access-99mvd\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.397073 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.396946    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-dshm\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.397073 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.396967    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-model-cache\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.397187 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.397080    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038f71e2-e6db-4549-984c-798dda6d667d-tls-certs\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.397187 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.397173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-tmp-dir\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.497781 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.497745    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-tmp-dir\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.497953 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.497789    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-home\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.497953 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.497810    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mvd\" (UniqueName: \"kubernetes.io/projected/038f71e2-e6db-4549-984c-798dda6d667d-kube-api-access-99mvd\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.497953 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.497834    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-dshm\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.497953 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.497859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-model-cache\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.497953 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.497904    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038f71e2-e6db-4549-984c-798dda6d667d-tls-certs\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.498227 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.498148    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-tmp-dir\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.498227 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.498201    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-home\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.498337 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.498292    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-model-cache\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.500184 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.500153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-dshm\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.500298 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.500281    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038f71e2-e6db-4549-984c-798dda6d667d-tls-certs\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.506507 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.506489    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mvd\" (UniqueName: \"kubernetes.io/projected/038f71e2-e6db-4549-984c-798dda6d667d-kube-api-access-99mvd\") pod \"rate-test-model-4f6333-kserve-7847476dd7-h98cm\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") " pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.601892 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.601849    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:27.734430 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:27.734359    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"]
Jul 14 18:12:27.736583 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:12:27.736552    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod038f71e2_e6db_4549_984c_798dda6d667d.slice/crio-97ebd8d5b3981d7c6266957baff4a9d5db931ed9089615130ff995c98d420b8a WatchSource:0}: Error finding container 97ebd8d5b3981d7c6266957baff4a9d5db931ed9089615130ff995c98d420b8a: Status 404 returned error can't find the container with id 97ebd8d5b3981d7c6266957baff4a9d5db931ed9089615130ff995c98d420b8a
Jul 14 18:12:28.291152 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:28.291115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm" event={"ID":"038f71e2-e6db-4549-984c-798dda6d667d","Type":"ContainerStarted","Data":"92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0"}
Jul 14 18:12:28.291152 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:28.291159    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm" event={"ID":"038f71e2-e6db-4549-984c-798dda6d667d","Type":"ContainerStarted","Data":"97ebd8d5b3981d7c6266957baff4a9d5db931ed9089615130ff995c98d420b8a"}
Jul 14 18:12:28.310249 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:28.310194    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm" podStartSLOduration=1.3101783519999999 podStartE2EDuration="1.310178352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:12:28.308954928 +0000 UTC m=+2794.853358724" watchObservedRunningTime="2026-07-14 18:12:28.310178352 +0000 UTC m=+2794.854582146"
Jul 14 18:12:37.602922 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:37.602885    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:37.603417 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:37.603146    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:37.615290 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:37.615256    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:12:38.344462 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:12:38.344433    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:13:14.296073 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:14.296037    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"]
Jul 14 18:13:21.363747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.363714    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:21.366518 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.366501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:21.369266 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.369249    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kdsgk\""
Jul 14 18:13:21.374144 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.374124    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:21.455443 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.455416    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkb9f\" (UniqueName: \"kubernetes.io/projected/957fca9f-6338-4d18-95d9-7af954c21c4e-kube-api-access-bkb9f\") pod \"test-curl-7417-140101077008432\" (UID: \"957fca9f-6338-4d18-95d9-7af954c21c4e\") " pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:21.556714 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.556692    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkb9f\" (UniqueName: \"kubernetes.io/projected/957fca9f-6338-4d18-95d9-7af954c21c4e-kube-api-access-bkb9f\") pod \"test-curl-7417-140101077008432\" (UID: \"957fca9f-6338-4d18-95d9-7af954c21c4e\") " pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:21.572195 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.572174    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkb9f\" (UniqueName: \"kubernetes.io/projected/957fca9f-6338-4d18-95d9-7af954c21c4e-kube-api-access-bkb9f\") pod \"test-curl-7417-140101077008432\" (UID: \"957fca9f-6338-4d18-95d9-7af954c21c4e\") " pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:21.675959 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.675904    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:21.805787 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.805750    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:21.806518 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:13:21.806489    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod957fca9f_6338_4d18_95d9_7af954c21c4e.slice/crio-fa31b22b5030d7294f5b45243fd305c47f881cce41c76fc5dfeaba09f6254f05 WatchSource:0}: Error finding container fa31b22b5030d7294f5b45243fd305c47f881cce41c76fc5dfeaba09f6254f05: Status 404 returned error can't find the container with id fa31b22b5030d7294f5b45243fd305c47f881cce41c76fc5dfeaba09f6254f05
Jul 14 18:13:21.808124 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:21.808107    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 18:13:22.530915 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:22.530888    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077008432" event={"ID":"957fca9f-6338-4d18-95d9-7af954c21c4e","Type":"ContainerStarted","Data":"fa31b22b5030d7294f5b45243fd305c47f881cce41c76fc5dfeaba09f6254f05"}
Jul 14 18:13:23.537176 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:23.537146    2567 generic.go:356] "Generic (PLEG): container finished" podID="957fca9f-6338-4d18-95d9-7af954c21c4e" containerID="714dd0fbada011288d7075c8f3811bec9b1fd43520331fa8db06f2a7b06999e9" exitCode=0
Jul 14 18:13:23.537563 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:23.537218    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077008432" event={"ID":"957fca9f-6338-4d18-95d9-7af954c21c4e","Type":"ContainerDied","Data":"714dd0fbada011288d7075c8f3811bec9b1fd43520331fa8db06f2a7b06999e9"}
Jul 14 18:13:24.671189 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.671165    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:24.705770 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.705748    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7417-140101077008432_957fca9f-6338-4d18-95d9-7af954c21c4e/test-curl-7417-140101077008432/0.log"
Jul 14 18:13:24.727026 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.726996    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:24.728493 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.728474    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:24.782904 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.782883    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkb9f\" (UniqueName: \"kubernetes.io/projected/957fca9f-6338-4d18-95d9-7af954c21c4e-kube-api-access-bkb9f\") pod \"957fca9f-6338-4d18-95d9-7af954c21c4e\" (UID: \"957fca9f-6338-4d18-95d9-7af954c21c4e\") "
Jul 14 18:13:24.784970 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.784939    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957fca9f-6338-4d18-95d9-7af954c21c4e-kube-api-access-bkb9f" (OuterVolumeSpecName: "kube-api-access-bkb9f") pod "957fca9f-6338-4d18-95d9-7af954c21c4e" (UID: "957fca9f-6338-4d18-95d9-7af954c21c4e"). InnerVolumeSpecName "kube-api-access-bkb9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:13:24.848115 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.848091    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:24.848454 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.848441    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957fca9f-6338-4d18-95d9-7af954c21c4e" containerName="test-curl-7417-140101077008432"
Jul 14 18:13:24.848508 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.848455    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="957fca9f-6338-4d18-95d9-7af954c21c4e" containerName="test-curl-7417-140101077008432"
Jul 14 18:13:24.848546 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.848528    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="957fca9f-6338-4d18-95d9-7af954c21c4e" containerName="test-curl-7417-140101077008432"
Jul 14 18:13:24.851369 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.851336    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:24.858171 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.858146    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:24.884296 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.884270    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pgp\" (UniqueName: \"kubernetes.io/projected/babd709c-a908-454f-8044-964901037f04-kube-api-access-x5pgp\") pod \"test-curl-7417-140101077008432\" (UID: \"babd709c-a908-454f-8044-964901037f04\") " pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:24.884396 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.884318    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkb9f\" (UniqueName: \"kubernetes.io/projected/957fca9f-6338-4d18-95d9-7af954c21c4e-kube-api-access-bkb9f\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:24.985174 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.985149    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pgp\" (UniqueName: \"kubernetes.io/projected/babd709c-a908-454f-8044-964901037f04-kube-api-access-x5pgp\") pod \"test-curl-7417-140101077008432\" (UID: \"babd709c-a908-454f-8044-964901037f04\") " pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:24.994530 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:24.994505    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pgp\" (UniqueName: \"kubernetes.io/projected/babd709c-a908-454f-8044-964901037f04-kube-api-access-x5pgp\") pod \"test-curl-7417-140101077008432\" (UID: \"babd709c-a908-454f-8044-964901037f04\") " pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:25.161470 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:25.161413    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:25.287595 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:25.287568    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:25.290339 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:13:25.290318    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbabd709c_a908_454f_8044_964901037f04.slice/crio-c58b56efc294b44890f0b5fbd46c82c1c6000e61357ef90792322cdf3b28b34d WatchSource:0}: Error finding container c58b56efc294b44890f0b5fbd46c82c1c6000e61357ef90792322cdf3b28b34d: Status 404 returned error can't find the container with id c58b56efc294b44890f0b5fbd46c82c1c6000e61357ef90792322cdf3b28b34d
Jul 14 18:13:25.547391 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:25.547304    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:25.547391 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:25.547322    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa31b22b5030d7294f5b45243fd305c47f881cce41c76fc5dfeaba09f6254f05"
Jul 14 18:13:25.548542 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:25.548511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077008432" event={"ID":"babd709c-a908-454f-8044-964901037f04","Type":"ContainerStarted","Data":"c58b56efc294b44890f0b5fbd46c82c1c6000e61357ef90792322cdf3b28b34d"}
Jul 14 18:13:25.558970 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:25.558942    2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7417-140101077008432" oldPodUID="957fca9f-6338-4d18-95d9-7af954c21c4e" podUID="babd709c-a908-454f-8044-964901037f04"
Jul 14 18:13:26.051651 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:26.051620    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957fca9f-6338-4d18-95d9-7af954c21c4e" path="/var/lib/kubelet/pods/957fca9f-6338-4d18-95d9-7af954c21c4e/volumes"
Jul 14 18:13:26.555328 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:26.555293    2567 generic.go:356] "Generic (PLEG): container finished" podID="babd709c-a908-454f-8044-964901037f04" containerID="6ef6e32699a81579c4f772c188ba01c1a48fcce499934a3f874179bc0d476f9b" exitCode=0
Jul 14 18:13:26.555527 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:26.555386    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077008432" event={"ID":"babd709c-a908-454f-8044-964901037f04","Type":"ContainerDied","Data":"6ef6e32699a81579c4f772c188ba01c1a48fcce499934a3f874179bc0d476f9b"}
Jul 14 18:13:27.686282 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.686258    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:27.706791 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.706769    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7417-140101077008432_babd709c-a908-454f-8044-964901037f04/test-curl-7417-140101077008432/0.log"
Jul 14 18:13:27.728841 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.728818    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:27.732328 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.732307    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-140101077008432"]
Jul 14 18:13:27.808598 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.808573    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5pgp\" (UniqueName: \"kubernetes.io/projected/babd709c-a908-454f-8044-964901037f04-kube-api-access-x5pgp\") pod \"babd709c-a908-454f-8044-964901037f04\" (UID: \"babd709c-a908-454f-8044-964901037f04\") "
Jul 14 18:13:27.810625 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.810579    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babd709c-a908-454f-8044-964901037f04-kube-api-access-x5pgp" (OuterVolumeSpecName: "kube-api-access-x5pgp") pod "babd709c-a908-454f-8044-964901037f04" (UID: "babd709c-a908-454f-8044-964901037f04"). InnerVolumeSpecName "kube-api-access-x5pgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:13:27.859463 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.859440    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:27.859780 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.859767    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babd709c-a908-454f-8044-964901037f04" containerName="test-curl-7417-140101077008432"
Jul 14 18:13:27.859826 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.859781    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="babd709c-a908-454f-8044-964901037f04" containerName="test-curl-7417-140101077008432"
Jul 14 18:13:27.859869 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.859833    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="babd709c-a908-454f-8044-964901037f04" containerName="test-curl-7417-140101077008432"
Jul 14 18:13:27.862664 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.862650    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:27.869444 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.869425    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:27.909761 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.909732    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldx8\" (UniqueName: \"kubernetes.io/projected/3719f3cb-f860-461c-ae74-f997836c9bbc-kube-api-access-rldx8\") pod \"test-curl-7417-140101077009440\" (UID: \"3719f3cb-f860-461c-ae74-f997836c9bbc\") " pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:27.909885 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:27.909786    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pgp\" (UniqueName: \"kubernetes.io/projected/babd709c-a908-454f-8044-964901037f04-kube-api-access-x5pgp\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:28.010820 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.010777    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldx8\" (UniqueName: \"kubernetes.io/projected/3719f3cb-f860-461c-ae74-f997836c9bbc-kube-api-access-rldx8\") pod \"test-curl-7417-140101077009440\" (UID: \"3719f3cb-f860-461c-ae74-f997836c9bbc\") " pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:28.019751 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.019718    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldx8\" (UniqueName: \"kubernetes.io/projected/3719f3cb-f860-461c-ae74-f997836c9bbc-kube-api-access-rldx8\") pod \"test-curl-7417-140101077009440\" (UID: \"3719f3cb-f860-461c-ae74-f997836c9bbc\") " pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:28.051771 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.051739    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babd709c-a908-454f-8044-964901037f04" path="/var/lib/kubelet/pods/babd709c-a908-454f-8044-964901037f04/volumes"
Jul 14 18:13:28.171837 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.171790    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:28.301657 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.301623    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:28.305232 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:13:28.305202    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3719f3cb_f860_461c_ae74_f997836c9bbc.slice/crio-cba37c8e4366a80cb8ab389fb5371577f147c09fdcfe94e0cc64c5c2a5397eed WatchSource:0}: Error finding container cba37c8e4366a80cb8ab389fb5371577f147c09fdcfe94e0cc64c5c2a5397eed: Status 404 returned error can't find the container with id cba37c8e4366a80cb8ab389fb5371577f147c09fdcfe94e0cc64c5c2a5397eed
Jul 14 18:13:28.565221 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.565133    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077008432"
Jul 14 18:13:28.565221 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.565148    2567 scope.go:117] "RemoveContainer" containerID="6ef6e32699a81579c4f772c188ba01c1a48fcce499934a3f874179bc0d476f9b"
Jul 14 18:13:28.566694 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:28.566669    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077009440" event={"ID":"3719f3cb-f860-461c-ae74-f997836c9bbc","Type":"ContainerStarted","Data":"cba37c8e4366a80cb8ab389fb5371577f147c09fdcfe94e0cc64c5c2a5397eed"}
Jul 14 18:13:29.315966 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.315919    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm" podUID="038f71e2-e6db-4549-984c-798dda6d667d" containerName="main" containerID="cri-o://92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0" gracePeriod=45
Jul 14 18:13:29.559942 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.559911    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:13:29.573531 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.573504    2567 generic.go:356] "Generic (PLEG): container finished" podID="3719f3cb-f860-461c-ae74-f997836c9bbc" containerID="aee935cc128b5fdc1fcd8e472a4835111790e6ca56f11c1f48bffc1997c3fdd0" exitCode=0
Jul 14 18:13:29.573644 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.573590    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077009440" event={"ID":"3719f3cb-f860-461c-ae74-f997836c9bbc","Type":"ContainerDied","Data":"aee935cc128b5fdc1fcd8e472a4835111790e6ca56f11c1f48bffc1997c3fdd0"}
Jul 14 18:13:29.574778 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.574760    2567 generic.go:356] "Generic (PLEG): container finished" podID="038f71e2-e6db-4549-984c-798dda6d667d" containerID="92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0" exitCode=0
Jul 14 18:13:29.574873 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.574787    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm" event={"ID":"038f71e2-e6db-4549-984c-798dda6d667d","Type":"ContainerDied","Data":"92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0"}
Jul 14 18:13:29.574873 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.574796    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"
Jul 14 18:13:29.574873 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.574810    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm" event={"ID":"038f71e2-e6db-4549-984c-798dda6d667d","Type":"ContainerDied","Data":"97ebd8d5b3981d7c6266957baff4a9d5db931ed9089615130ff995c98d420b8a"}
Jul 14 18:13:29.574873 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.574826    2567 scope.go:117] "RemoveContainer" containerID="92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0"
Jul 14 18:13:29.584487 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.584470    2567 scope.go:117] "RemoveContainer" containerID="92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0"
Jul 14 18:13:29.584748 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:13:29.584730    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0\": container with ID starting with 92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0 not found: ID does not exist" containerID="92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0"
Jul 14 18:13:29.584811 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.584759    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0"} err="failed to get container status \"92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0\": rpc error: code = NotFound desc = could not find container \"92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0\": container with ID starting with 92331b5a27d0a3105510bd2d3ad0fafcaf3db482ae7d6398215f5448a65fa6a0 not found: ID does not exist"
Jul 14 18:13:29.626094 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626072    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-model-cache\") pod \"038f71e2-e6db-4549-984c-798dda6d667d\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") "
Jul 14 18:13:29.626220 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626139    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-home\") pod \"038f71e2-e6db-4549-984c-798dda6d667d\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") "
Jul 14 18:13:29.626220 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626177    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-tmp-dir\") pod \"038f71e2-e6db-4549-984c-798dda6d667d\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") "
Jul 14 18:13:29.626220 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626207    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-dshm\") pod \"038f71e2-e6db-4549-984c-798dda6d667d\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") "
Jul 14 18:13:29.626399 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626248    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038f71e2-e6db-4549-984c-798dda6d667d-tls-certs\") pod \"038f71e2-e6db-4549-984c-798dda6d667d\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") "
Jul 14 18:13:29.626399 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626275    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99mvd\" (UniqueName: \"kubernetes.io/projected/038f71e2-e6db-4549-984c-798dda6d667d-kube-api-access-99mvd\") pod \"038f71e2-e6db-4549-984c-798dda6d667d\" (UID: \"038f71e2-e6db-4549-984c-798dda6d667d\") "
Jul 14 18:13:29.626399 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626371    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-home" (OuterVolumeSpecName: "home") pod "038f71e2-e6db-4549-984c-798dda6d667d" (UID: "038f71e2-e6db-4549-984c-798dda6d667d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:13:29.626530 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626388    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-model-cache" (OuterVolumeSpecName: "model-cache") pod "038f71e2-e6db-4549-984c-798dda6d667d" (UID: "038f71e2-e6db-4549-984c-798dda6d667d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:13:29.626530 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626468    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "038f71e2-e6db-4549-984c-798dda6d667d" (UID: "038f71e2-e6db-4549-984c-798dda6d667d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:13:29.626660 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626642    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-model-cache\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:29.626660 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626659    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-home\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:29.626838 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.626684    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-tmp-dir\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:29.628274 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.628242    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038f71e2-e6db-4549-984c-798dda6d667d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "038f71e2-e6db-4549-984c-798dda6d667d" (UID: "038f71e2-e6db-4549-984c-798dda6d667d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 18:13:29.628378 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.628271    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038f71e2-e6db-4549-984c-798dda6d667d-kube-api-access-99mvd" (OuterVolumeSpecName: "kube-api-access-99mvd") pod "038f71e2-e6db-4549-984c-798dda6d667d" (UID: "038f71e2-e6db-4549-984c-798dda6d667d"). InnerVolumeSpecName "kube-api-access-99mvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:13:29.628378 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.628247    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-dshm" (OuterVolumeSpecName: "dshm") pod "038f71e2-e6db-4549-984c-798dda6d667d" (UID: "038f71e2-e6db-4549-984c-798dda6d667d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:13:29.727888 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.727851    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/038f71e2-e6db-4549-984c-798dda6d667d-dshm\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:29.727888 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.727887    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/038f71e2-e6db-4549-984c-798dda6d667d-tls-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:29.728051 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.727902    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99mvd\" (UniqueName: \"kubernetes.io/projected/038f71e2-e6db-4549-984c-798dda6d667d-kube-api-access-99mvd\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:29.897379 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.897334    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"]
Jul 14 18:13:29.900236 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:29.900212    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4f6333/rate-test-model-4f6333-kserve-7847476dd7-h98cm"]
Jul 14 18:13:30.052733 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.052703    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="038f71e2-e6db-4549-984c-798dda6d667d" path="/var/lib/kubelet/pods/038f71e2-e6db-4549-984c-798dda6d667d/volumes"
Jul 14 18:13:30.708251 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.708224    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:30.727777 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.727751    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7417-140101077009440_3719f3cb-f860-461c-ae74-f997836c9bbc/test-curl-7417-140101077009440/0.log"
Jul 14 18:13:30.751730 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.751699    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:30.753990 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.753966    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:30.837019 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.836926    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rldx8\" (UniqueName: \"kubernetes.io/projected/3719f3cb-f860-461c-ae74-f997836c9bbc-kube-api-access-rldx8\") pod \"3719f3cb-f860-461c-ae74-f997836c9bbc\" (UID: \"3719f3cb-f860-461c-ae74-f997836c9bbc\") "
Jul 14 18:13:30.839007 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.838979    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3719f3cb-f860-461c-ae74-f997836c9bbc-kube-api-access-rldx8" (OuterVolumeSpecName: "kube-api-access-rldx8") pod "3719f3cb-f860-461c-ae74-f997836c9bbc" (UID: "3719f3cb-f860-461c-ae74-f997836c9bbc"). InnerVolumeSpecName "kube-api-access-rldx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:13:30.870833 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.870809    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:30.871156 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.871144    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="038f71e2-e6db-4549-984c-798dda6d667d" containerName="main"
Jul 14 18:13:30.871204 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.871159    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="038f71e2-e6db-4549-984c-798dda6d667d" containerName="main"
Jul 14 18:13:30.871204 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.871167    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3719f3cb-f860-461c-ae74-f997836c9bbc" containerName="test-curl-7417-140101077009440"
Jul 14 18:13:30.871204 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.871173    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3719f3cb-f860-461c-ae74-f997836c9bbc" containerName="test-curl-7417-140101077009440"
Jul 14 18:13:30.871299 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.871221    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3719f3cb-f860-461c-ae74-f997836c9bbc" containerName="test-curl-7417-140101077009440"
Jul 14 18:13:30.871299 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.871229    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="038f71e2-e6db-4549-984c-798dda6d667d" containerName="main"
Jul 14 18:13:30.874069 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.874051    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:30.881225 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.881204    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:30.938544 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.938521    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/1146e7e4-812d-4827-b5d8-5cc9262b7353-kube-api-access-psvxq\") pod \"test-curl-7417-140101077009440\" (UID: \"1146e7e4-812d-4827-b5d8-5cc9262b7353\") " pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:30.938658 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:30.938633    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rldx8\" (UniqueName: \"kubernetes.io/projected/3719f3cb-f860-461c-ae74-f997836c9bbc-kube-api-access-rldx8\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:31.039636 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.039610    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/1146e7e4-812d-4827-b5d8-5cc9262b7353-kube-api-access-psvxq\") pod \"test-curl-7417-140101077009440\" (UID: \"1146e7e4-812d-4827-b5d8-5cc9262b7353\") " pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:31.048086 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.048063    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/1146e7e4-812d-4827-b5d8-5cc9262b7353-kube-api-access-psvxq\") pod \"test-curl-7417-140101077009440\" (UID: \"1146e7e4-812d-4827-b5d8-5cc9262b7353\") " pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:31.183547 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.183455    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:31.307937 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.307907    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:31.310737 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:13:31.310711    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1146e7e4_812d_4827_b5d8_5cc9262b7353.slice/crio-d9bc6930bd68c71164a3404fbc318c0bb859725f1f3e01bc6812f9cb5ed57d92 WatchSource:0}: Error finding container d9bc6930bd68c71164a3404fbc318c0bb859725f1f3e01bc6812f9cb5ed57d92: Status 404 returned error can't find the container with id d9bc6930bd68c71164a3404fbc318c0bb859725f1f3e01bc6812f9cb5ed57d92
Jul 14 18:13:31.585709 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.585674    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077009440" event={"ID":"1146e7e4-812d-4827-b5d8-5cc9262b7353","Type":"ContainerStarted","Data":"d9bc6930bd68c71164a3404fbc318c0bb859725f1f3e01bc6812f9cb5ed57d92"}
Jul 14 18:13:31.586817 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.586800    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:31.586817 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.586814    2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba37c8e4366a80cb8ab389fb5371577f147c09fdcfe94e0cc64c5c2a5397eed"
Jul 14 18:13:31.598674 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:31.598648    2567 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7417-140101077009440" oldPodUID="3719f3cb-f860-461c-ae74-f997836c9bbc" podUID="1146e7e4-812d-4827-b5d8-5cc9262b7353"
Jul 14 18:13:32.052096 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:32.052022    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3719f3cb-f860-461c-ae74-f997836c9bbc" path="/var/lib/kubelet/pods/3719f3cb-f860-461c-ae74-f997836c9bbc/volumes"
Jul 14 18:13:32.593276 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:32.593246    2567 generic.go:356] "Generic (PLEG): container finished" podID="1146e7e4-812d-4827-b5d8-5cc9262b7353" containerID="d1824ff27a0da8248c27a72651f97e87ed3ea6f49b0033b5c5484edf2e3930d7" exitCode=0
Jul 14 18:13:32.593276 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:32.593279    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-140101077009440" event={"ID":"1146e7e4-812d-4827-b5d8-5cc9262b7353","Type":"ContainerDied","Data":"d1824ff27a0da8248c27a72651f97e87ed3ea6f49b0033b5c5484edf2e3930d7"}
Jul 14 18:13:33.727568 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.727546    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:33.747941 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.747890    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7417-140101077009440_1146e7e4-812d-4827-b5d8-5cc9262b7353/test-curl-7417-140101077009440/0.log"
Jul 14 18:13:33.765331 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.765305    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/1146e7e4-812d-4827-b5d8-5cc9262b7353-kube-api-access-psvxq\") pod \"1146e7e4-812d-4827-b5d8-5cc9262b7353\" (UID: \"1146e7e4-812d-4827-b5d8-5cc9262b7353\") "
Jul 14 18:13:33.767813 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.767785    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1146e7e4-812d-4827-b5d8-5cc9262b7353-kube-api-access-psvxq" (OuterVolumeSpecName: "kube-api-access-psvxq") pod "1146e7e4-812d-4827-b5d8-5cc9262b7353" (UID: "1146e7e4-812d-4827-b5d8-5cc9262b7353"). InnerVolumeSpecName "kube-api-access-psvxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:13:33.768902 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.768880    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:33.770410 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.770394    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-140101077009440"]
Jul 14 18:13:33.866221 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:33.866151    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/1146e7e4-812d-4827-b5d8-5cc9262b7353-kube-api-access-psvxq\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:13:34.052524 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:34.052487    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1146e7e4-812d-4827-b5d8-5cc9262b7353" path="/var/lib/kubelet/pods/1146e7e4-812d-4827-b5d8-5cc9262b7353/volumes"
Jul 14 18:13:34.603722 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:34.603685    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-140101077009440"
Jul 14 18:13:34.603722 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:13:34.603714    2567 scope.go:117] "RemoveContainer" containerID="d1824ff27a0da8248c27a72651f97e87ed3ea6f49b0033b5c5484edf2e3930d7"
Jul 14 18:14:25.834277 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:25.834243    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"]
Jul 14 18:14:25.834900 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:25.834541    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy" containerID="cri-o://727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139" gracePeriod=30
Jul 14 18:14:26.549711 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:26.549679    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cdf9fb9f4-j9zlc_177b6848-394f-4ade-885f-428922cd85d9/authorino/0.log"
Jul 14 18:14:31.123590 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.123568    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:14:31.261964 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.261932    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istiod-ca-cert\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262120 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262005    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-credential-socket\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262120 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262028    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-envoy\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262265 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262148    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-976lc\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-kube-api-access-976lc\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262265 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262208    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-socket\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262265 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262249    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-podinfo\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262521 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262306    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-token\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262521 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262335    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-data\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262521 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262338    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:14:31.262521 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262375    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 18:14:31.262521 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262397    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-certs\") pod \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\" (UID: \"5d6b21a6-0e5e-4109-8612-41d7a420f9c6\") "
Jul 14 18:14:31.262762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262690    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:14:31.262762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262711    2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-credential-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.262762 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262730    2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istiod-ca-cert\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.262875 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.262776    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:14:31.263286 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.263259    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:14:31.264268 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.264247    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 18:14:31.264499 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.264478    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-kube-api-access-976lc" (OuterVolumeSpecName: "kube-api-access-976lc") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "kube-api-access-976lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:14:31.264720 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.264699    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 18:14:31.264786 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.264754    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d6b21a6-0e5e-4109-8612-41d7a420f9c6" (UID: "5d6b21a6-0e5e-4109-8612-41d7a420f9c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363923    2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-token\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363949    2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-data\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363959    2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-certs\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363966    2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-envoy\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363975    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976lc\" (UniqueName: \"kubernetes.io/projected/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-kube-api-access-976lc\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363983    2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-workload-socket\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.363999 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.363992    2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6b21a6-0e5e-4109-8612-41d7a420f9c6-istio-podinfo\") on node \"ip-10-0-138-98.ec2.internal\" DevicePath \"\""
Jul 14 18:14:31.585237 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.585200    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-42b9r_316a6d30-910a-441e-9d41-dd36d556ee68/manager/0.log"
Jul 14 18:14:31.877673 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.877647    2567 generic.go:356] "Generic (PLEG): container finished" podID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerID="727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139" exitCode=0
Jul 14 18:14:31.877842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.877711    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"
Jul 14 18:14:31.877842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.877720    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" event={"ID":"5d6b21a6-0e5e-4109-8612-41d7a420f9c6","Type":"ContainerDied","Data":"727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139"}
Jul 14 18:14:31.877842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.877785    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf" event={"ID":"5d6b21a6-0e5e-4109-8612-41d7a420f9c6","Type":"ContainerDied","Data":"b15e1acee0aa15e231b9c3823e018332501806a2220575e4efd70a5dea4fea1f"}
Jul 14 18:14:31.877842 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.877808    2567 scope.go:117] "RemoveContainer" containerID="727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139"
Jul 14 18:14:31.886794 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.886747    2567 scope.go:117] "RemoveContainer" containerID="22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"
Jul 14 18:14:31.894200 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.894184    2567 scope.go:117] "RemoveContainer" containerID="727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139"
Jul 14 18:14:31.894463 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:14:31.894443    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139\": container with ID starting with 727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139 not found: ID does not exist" containerID="727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139"
Jul 14 18:14:31.894554 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.894475    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139"} err="failed to get container status \"727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139\": rpc error: code = NotFound desc = could not find container \"727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139\": container with ID starting with 727b8de4d148880ba7bbbabdb60626ad0e4433f98a5d197e1b82eeb9f89b7139 not found: ID does not exist"
Jul 14 18:14:31.894554 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.894499    2567 scope.go:117] "RemoveContainer" containerID="22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"
Jul 14 18:14:31.894732 ip-10-0-138-98 kubenswrapper[2567]: E0714 18:14:31.894714    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4\": container with ID starting with 22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4 not found: ID does not exist" containerID="22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"
Jul 14 18:14:31.894768 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.894738    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4"} err="failed to get container status \"22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4\": rpc error: code = NotFound desc = could not find container \"22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4\": container with ID starting with 22ecaedb8ae2425b5f445c7ec27eb2b2aa5aaeadf83a5e099296ef0fbf4552f4 not found: ID does not exist"
Jul 14 18:14:31.900666 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.900646    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"]
Jul 14 18:14:31.904127 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:31.904106    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4f6333-openshift-default-7b44f6c5-cr2pf"]
Jul 14 18:14:32.051544 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:32.051516    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" path="/var/lib/kubelet/pods/5d6b21a6-0e5e-4109-8612-41d7a420f9c6/volumes"
Jul 14 18:14:32.085052 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:32.085021    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f54d68b84-glz67_1dcf83b9-cc2d-466b-be3d-f6bfe1757234/copy-manifests/0.log"
Jul 14 18:14:32.092497 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:32.092479    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f54d68b84-glz67_1dcf83b9-cc2d-466b-be3d-f6bfe1757234/manager/0.log"
Jul 14 18:14:35.642448 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:35.642415    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cdf9fb9f4-j9zlc_177b6848-394f-4ade-885f-428922cd85d9/authorino/0.log"
Jul 14 18:14:36.457785 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:36.457759    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dhmrh_2521423d-6155-4be1-b344-35d5cbecbd62/manager/0.log"
Jul 14 18:14:36.806246 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:36.806167    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678dd9xp_fcab530e-7cac-465f-93dc-4ea18db00fca/istio-proxy/0.log"
Jul 14 18:14:37.034149 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:37.034115    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfddfddb9-s6qvh_745389ea-c983-4104-b107-09fcf138d169/kube-auth-proxy/0.log"
Jul 14 18:14:38.431014 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:38.430986    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-77czt_dae2cca9-3da7-426f-887f-1e4c3e6cda8c/main/0.log"
Jul 14 18:14:38.548991 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:38.548962    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-59pbm_fe033f73-aa97-4050-bbca-0a0e5277eb6b/main/0.log"
Jul 14 18:14:42.301891 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.301849    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4c4r8/must-gather-24t4k"]
Jul 14 18:14:42.302367 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302335    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy"
Jul 14 18:14:42.302454 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302370    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy"
Jul 14 18:14:42.302454 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302386    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy"
Jul 14 18:14:42.302454 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302395    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy"
Jul 14 18:14:42.302454 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302404    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1146e7e4-812d-4827-b5d8-5cc9262b7353" containerName="test-curl-7417-140101077009440"
Jul 14 18:14:42.302454 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302444    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="1146e7e4-812d-4827-b5d8-5cc9262b7353" containerName="test-curl-7417-140101077009440"
Jul 14 18:14:42.302706 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302569    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy"
Jul 14 18:14:42.302706 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302585    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="1146e7e4-812d-4827-b5d8-5cc9262b7353" containerName="test-curl-7417-140101077009440"
Jul 14 18:14:42.302808 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.302744    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6b21a6-0e5e-4109-8612-41d7a420f9c6" containerName="istio-proxy"
Jul 14 18:14:42.305716 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.305697    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.308556 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.308533    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4c4r8\"/\"openshift-service-ca.crt\""
Jul 14 18:14:42.308811 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.308795    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4c4r8\"/\"kube-root-ca.crt\""
Jul 14 18:14:42.308859 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.308838    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4c4r8\"/\"default-dockercfg-vpwkq\""
Jul 14 18:14:42.316100 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.316079    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4c4r8/must-gather-24t4k"]
Jul 14 18:14:42.458408 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.458385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6gb\" (UniqueName: \"kubernetes.io/projected/0745c1d9-3ac6-4bda-b60b-100c32d0c643-kube-api-access-zs6gb\") pod \"must-gather-24t4k\" (UID: \"0745c1d9-3ac6-4bda-b60b-100c32d0c643\") " pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.458522 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.458431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0745c1d9-3ac6-4bda-b60b-100c32d0c643-must-gather-output\") pod \"must-gather-24t4k\" (UID: \"0745c1d9-3ac6-4bda-b60b-100c32d0c643\") " pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.559430 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.559368    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6gb\" (UniqueName: \"kubernetes.io/projected/0745c1d9-3ac6-4bda-b60b-100c32d0c643-kube-api-access-zs6gb\") pod \"must-gather-24t4k\" (UID: \"0745c1d9-3ac6-4bda-b60b-100c32d0c643\") " pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.559430 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.559412    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0745c1d9-3ac6-4bda-b60b-100c32d0c643-must-gather-output\") pod \"must-gather-24t4k\" (UID: \"0745c1d9-3ac6-4bda-b60b-100c32d0c643\") " pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.559801 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.559779    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0745c1d9-3ac6-4bda-b60b-100c32d0c643-must-gather-output\") pod \"must-gather-24t4k\" (UID: \"0745c1d9-3ac6-4bda-b60b-100c32d0c643\") " pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.568125 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.568104    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6gb\" (UniqueName: \"kubernetes.io/projected/0745c1d9-3ac6-4bda-b60b-100c32d0c643-kube-api-access-zs6gb\") pod \"must-gather-24t4k\" (UID: \"0745c1d9-3ac6-4bda-b60b-100c32d0c643\") " pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.614697 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.614673    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4c4r8/must-gather-24t4k"
Jul 14 18:14:42.748262 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.748238    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4c4r8/must-gather-24t4k"]
Jul 14 18:14:42.750166 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:14:42.750139    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0745c1d9_3ac6_4bda_b60b_100c32d0c643.slice/crio-1b94ab15de78b4b96294651f2c462d243ac6df3cd4e40b41564edbd415f9fcc9 WatchSource:0}: Error finding container 1b94ab15de78b4b96294651f2c462d243ac6df3cd4e40b41564edbd415f9fcc9: Status 404 returned error can't find the container with id 1b94ab15de78b4b96294651f2c462d243ac6df3cd4e40b41564edbd415f9fcc9
Jul 14 18:14:42.933861 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:42.933833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4c4r8/must-gather-24t4k" event={"ID":"0745c1d9-3ac6-4bda-b60b-100c32d0c643","Type":"ContainerStarted","Data":"1b94ab15de78b4b96294651f2c462d243ac6df3cd4e40b41564edbd415f9fcc9"}
Jul 14 18:14:43.940936 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:43.940893    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4c4r8/must-gather-24t4k" event={"ID":"0745c1d9-3ac6-4bda-b60b-100c32d0c643","Type":"ContainerStarted","Data":"e5e98826c6f206c16df3d9926ae31f6ea8a1d4d6a83ac260306316c3146c8b22"}
Jul 14 18:14:43.940936 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:43.940937    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4c4r8/must-gather-24t4k" event={"ID":"0745c1d9-3ac6-4bda-b60b-100c32d0c643","Type":"ContainerStarted","Data":"898d8c830e4467d48e45948b193887916a52964b60762badb8db4dceb64980b5"}
Jul 14 18:14:43.958474 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:43.958424    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4c4r8/must-gather-24t4k" podStartSLOduration=1.206789646 podStartE2EDuration="1.958407709s" totalImagesPullingTime="751.618063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:43.957540952 +0000 UTC m=+2930.501944750" watchObservedRunningTime="2026-07-14 18:14:43.958407709 +0000 UTC m=+2930.502811503"
Jul 14 18:14:45.027962 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:45.027926    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-69hq7_f76fad34-a307-43b9-b220-75c29a93352b/global-pull-secret-syncer/0.log"
Jul 14 18:14:45.255460 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:45.255430    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mhnz7_c7ec9dfb-31b6-4806-92f4-6c36debc59d3/konnectivity-agent/0.log"
Jul 14 18:14:45.306895 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:45.306803    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-98.ec2.internal_8c5765f711f78dde18a902a1fe60d146/haproxy/0.log"
Jul 14 18:14:50.693246 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:50.693145    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cdf9fb9f4-j9zlc_177b6848-394f-4ade-885f-428922cd85d9/authorino/0.log"
Jul 14 18:14:51.105196 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:51.105149    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-dhmrh_2521423d-6155-4be1-b344-35d5cbecbd62/manager/0.log"
Jul 14 18:14:53.016376 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.016276    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-tqrqg_29d76197-e02d-4953-ac8c-08c45813a7fe/kube-state-metrics/0.log"
Jul 14 18:14:53.038175 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.038141    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-tqrqg_29d76197-e02d-4953-ac8c-08c45813a7fe/kube-rbac-proxy-main/0.log"
Jul 14 18:14:53.061851 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.061826    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-tqrqg_29d76197-e02d-4953-ac8c-08c45813a7fe/kube-rbac-proxy-self/0.log"
Jul 14 18:14:53.087708 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.087677    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68cc46949c-2kn2s_2dee9203-7020-4600-a8e6-898ca56d7411/metrics-server/0.log"
Jul 14 18:14:53.113068 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.113031    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-r7575_055a5c57-6812-44e4-995c-3d645338393f/monitoring-plugin/0.log"
Jul 14 18:14:53.301679 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.301590    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qpbk9_208c5abd-9e61-45ba-8575-69fdfa665e5b/node-exporter/0.log"
Jul 14 18:14:53.321712 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.321686    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qpbk9_208c5abd-9e61-45ba-8575-69fdfa665e5b/kube-rbac-proxy/0.log"
Jul 14 18:14:53.344116 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.344093    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qpbk9_208c5abd-9e61-45ba-8575-69fdfa665e5b/init-textfile/0.log"
Jul 14 18:14:53.373407 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.373382    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-lb8kv_6a6d3c32-51aa-42c7-acfb-d1c804cf783a/kube-rbac-proxy-main/0.log"
Jul 14 18:14:53.396646 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.396623    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-lb8kv_6a6d3c32-51aa-42c7-acfb-d1c804cf783a/kube-rbac-proxy-self/0.log"
Jul 14 18:14:53.417365 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.417317    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-lb8kv_6a6d3c32-51aa-42c7-acfb-d1c804cf783a/openshift-state-metrics/0.log"
Jul 14 18:14:53.471244 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.471209    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/prometheus/0.log"
Jul 14 18:14:53.493087 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.493059    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/config-reloader/0.log"
Jul 14 18:14:53.514601 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.514572    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/thanos-sidecar/0.log"
Jul 14 18:14:53.537145 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.537121    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/kube-rbac-proxy-web/0.log"
Jul 14 18:14:53.563004 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.562936    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/kube-rbac-proxy/0.log"
Jul 14 18:14:53.592573 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.592539    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/kube-rbac-proxy-thanos/0.log"
Jul 14 18:14:53.620717 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.620693    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6583dc36-edf9-4496-9e33-c21d01944044/init-config-reloader/0.log"
Jul 14 18:14:53.716033 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.716000    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d46c5b8c-4scmj_910ebf57-1b42-4a8d-9170-0ee64f399cd1/telemeter-client/0.log"
Jul 14 18:14:53.737253 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.737229    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d46c5b8c-4scmj_910ebf57-1b42-4a8d-9170-0ee64f399cd1/reload/0.log"
Jul 14 18:14:53.758450 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:53.758416    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d46c5b8c-4scmj_910ebf57-1b42-4a8d-9170-0ee64f399cd1/kube-rbac-proxy/0.log"
Jul 14 18:14:54.995304 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:54.995272    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"]
Jul 14 18:14:54.999947 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:54.999922    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.009018 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.008833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"]
Jul 14 18:14:55.098665 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.098620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-sys\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.098856 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.098724    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-podres\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.098856 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.098771    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccwdb\" (UniqueName: \"kubernetes.io/projected/fc985f31-c9ea-4296-bc56-dfa949f8f16d-kube-api-access-ccwdb\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.098986 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.098856    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-lib-modules\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.098986 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.098885    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-proc\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199363 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-podres\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199535 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199393    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccwdb\" (UniqueName: \"kubernetes.io/projected/fc985f31-c9ea-4296-bc56-dfa949f8f16d-kube-api-access-ccwdb\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199535 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-lib-modules\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199535 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-podres\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199535 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-proc\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199536    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-sys\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199553    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-proc\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199585    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-lib-modules\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.199747 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.199643    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc985f31-c9ea-4296-bc56-dfa949f8f16d-sys\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.207792 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.207768    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccwdb\" (UniqueName: \"kubernetes.io/projected/fc985f31-c9ea-4296-bc56-dfa949f8f16d-kube-api-access-ccwdb\") pod \"perf-node-gather-daemonset-6csws\" (UID: \"fc985f31-c9ea-4296-bc56-dfa949f8f16d\") " pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.314261 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.314178    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:55.497439 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:55.495172    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"]
Jul 14 18:14:55.499847 ip-10-0-138-98 kubenswrapper[2567]: W0714 18:14:55.499791    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfc985f31_c9ea_4296_bc56_dfa949f8f16d.slice/crio-c08ac03c1f9ea723f2a6bb94b424f2067b88f59ac1589df69275d544b5f597ad WatchSource:0}: Error finding container c08ac03c1f9ea723f2a6bb94b424f2067b88f59ac1589df69275d544b5f597ad: Status 404 returned error can't find the container with id c08ac03c1f9ea723f2a6bb94b424f2067b88f59ac1589df69275d544b5f597ad
Jul 14 18:14:56.022511 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:56.022477    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws" event={"ID":"fc985f31-c9ea-4296-bc56-dfa949f8f16d","Type":"ContainerStarted","Data":"4bfc05b2397df4523f1ae5e57f2501545fb5d05dea989e918fec261bab19f4c3"}
Jul 14 18:14:56.023009 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:56.022530    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws" event={"ID":"fc985f31-c9ea-4296-bc56-dfa949f8f16d","Type":"ContainerStarted","Data":"c08ac03c1f9ea723f2a6bb94b424f2067b88f59ac1589df69275d544b5f597ad"}
Jul 14 18:14:56.023009 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:56.022602    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:14:56.038574 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:56.038516    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws" podStartSLOduration=2.038493441 podStartE2EDuration="2.038493441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:14:56.037478575 +0000 UTC m=+2942.581882384" watchObservedRunningTime="2026-07-14 18:14:56.038493441 +0000 UTC m=+2942.582897238"
Jul 14 18:14:57.830846 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:57.830813    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fgtjn_ad73822d-c866-4d05-bb99-c05be31ca85f/dns/0.log"
Jul 14 18:14:57.851976 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:57.851949    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fgtjn_ad73822d-c866-4d05-bb99-c05be31ca85f/kube-rbac-proxy/0.log"
Jul 14 18:14:57.975180 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:57.975153    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8hg9g_223f6af3-059f-4613-b60b-787215b37707/dns-node-resolver/0.log"
Jul 14 18:14:58.557603 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:58.557573    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64f56f78d5-gcxvq_b7a0cd7b-45ab-439f-9fba-962130dc2720/registry/0.log"
Jul 14 18:14:58.633063 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:58.633015    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qwwj2_3b2853cf-fd5f-46a9-94a9-e73045273824/node-ca/0.log"
Jul 14 18:14:59.549484 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:59.549457    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678dd9xp_fcab530e-7cac-465f-93dc-4ea18db00fca/istio-proxy/0.log"
Jul 14 18:14:59.684849 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:14:59.684818    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfddfddb9-s6qvh_745389ea-c983-4104-b107-09fcf138d169/kube-auth-proxy/0.log"
Jul 14 18:15:00.621165 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:00.621138    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p8vrm_c5c36d5c-171f-4521-9568-6865dcf9c8e6/serve-healthcheck-canary/0.log"
Jul 14 18:15:01.144557 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:01.144525    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79l4c_c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897/kube-rbac-proxy/0.log"
Jul 14 18:15:01.164525 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:01.164499    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79l4c_c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897/exporter/0.log"
Jul 14 18:15:01.185125 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:01.185098    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-79l4c_c0abbd0d-6aaa-45c3-9e4e-b44c14ca7897/extractor/0.log"
Jul 14 18:15:02.041025 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:02.040999    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4c4r8/perf-node-gather-daemonset-6csws"
Jul 14 18:15:03.667841 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:03.667810    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-42b9r_316a6d30-910a-441e-9d41-dd36d556ee68/manager/0.log"
Jul 14 18:15:04.003673 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:04.003641    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f54d68b84-glz67_1dcf83b9-cc2d-466b-be3d-f6bfe1757234/manager/0.log"
Jul 14 18:15:04.029861 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:04.029821    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f54d68b84-glz67_1dcf83b9-cc2d-466b-be3d-f6bfe1757234/copy-manifests/0.log"
Jul 14 18:15:05.588915 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:05.588886    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cb94d9664-qbq9j_d84d2662-bafb-4ade-983a-2893c99cf79d/manager/0.log"
Jul 14 18:15:05.902107 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:05.902014    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678dd9xp_fcab530e-7cac-465f-93dc-4ea18db00fca/istio-proxy/0.log"
Jul 14 18:15:06.037220 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:06.037189    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfddfddb9-s6qvh_745389ea-c983-4104-b107-09fcf138d169/kube-auth-proxy/0.log"
Jul 14 18:15:07.335724 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:07.335688    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678dd9xp_fcab530e-7cac-465f-93dc-4ea18db00fca/istio-proxy/0.log"
Jul 14 18:15:07.464276 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:07.464245    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dfddfddb9-s6qvh_745389ea-c983-4104-b107-09fcf138d169/kube-auth-proxy/0.log"
Jul 14 18:15:11.892714 ip-10-0-138-98 kubenswrapper[2567]: I0714 18:15:11.892681    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5rjcc_acbf3934-a6a1-49a6-b56e-edcd941ee2d8/kube-multus/0.log"