Jul 27 16:17:01.223024 ip-10-0-129-154 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 16:17:01.597896 ip-10-0-129-154 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:17:01.597896 ip-10-0-129-154 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 16:17:01.597896 ip-10-0-129-154 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:17:01.597896 ip-10-0-129-154 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 16:17:01.597896 ip-10-0-129-154 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 16:17:01.598800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.598669    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 16:17:01.605927 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605895    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:17:01.605927 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605923    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:17:01.605927 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605930    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605935    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605939    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605942    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605945    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605947    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605955    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605958    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605962    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605965    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605968    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605970    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605973    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605975    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605978    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605981    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605983    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605986    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605988    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605991    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:17:01.606048 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605993    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.605996    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606000    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606004    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606007    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606010    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606012    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606015    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606018    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606020    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606023    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606026    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606028    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606030    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606034    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606037    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606039    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606042    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606045    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:17:01.606560 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606048    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606050    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606053    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606055    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606058    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606061    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606063    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606066    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606068    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606071    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606073    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606076    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606078    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606080    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606083    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606086    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606088    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606092    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606096    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:17:01.607053 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606099    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606101    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606104    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606106    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606109    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606111    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606115    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606117    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606121    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606124    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606127    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606130    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606132    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606135    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606137    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606140    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606143    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606145    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606149    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606151    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:17:01.607514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606154    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606157    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606159    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606163    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606165    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606168    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606170    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606173    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606176    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606178    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606181    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606183    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606186    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606188    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606191    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606194    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606196    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606199    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606201    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.606993    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:17:01.608035 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607002    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607005    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607008    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607010    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607014    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607018    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607022    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607026    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607030    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607033    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607036    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607039    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607042    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607045    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607048    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607050    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607053    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607055    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607058    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:17:01.608550 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607061    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607063    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607066    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607068    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607071    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607074    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607076    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607078    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607081    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607083    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607085    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607088    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607090    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607094    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607097    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607101    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607104    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607107    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607111    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607114    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:17:01.609088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607116    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607119    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607121    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607123    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607127    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607129    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607132    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607134    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607136    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607139    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607141    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607144    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607146    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607149    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607151    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607154    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607157    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607160    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607162    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607165    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:17:01.609581 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607168    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607170    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607173    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607176    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607178    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607181    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607184    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607186    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607189    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607191    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607194    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607204    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607207    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607210    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607212    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607214    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607217    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607220    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607222    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607224    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:17:01.610099 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607227    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607230    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607232    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607236    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607238    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607241    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607244    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607246    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607249    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607251    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607254    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607257    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607260    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607263    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607265    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607268    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607270    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607273    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607276    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:17:01.610595 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607720    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607726    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607728    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607731    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607734    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607737    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607740    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607742    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607745    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607747    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607749    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607752    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607754    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607757    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607760    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607762    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607765    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607768    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607771    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:17:01.611080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607773    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607776    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607778    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607781    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607783    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607786    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607789    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607791    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607794    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607797    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607799    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607802    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607804    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607807    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607810    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607812    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607814    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607817    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607820    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607823    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607825    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:17:01.611537 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607828    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607830    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607833    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607835    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607837    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607840    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607842    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607845    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607847    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607850    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607854    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607856    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607859    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607861    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607864    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607866    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607868    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607871    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607873    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607875    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:17:01.612064 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607878    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607880    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607882    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607886    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607890    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607893    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607895    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607898    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607900    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607903    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607906    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607909    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607911    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607914    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607916    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607919    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607921    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607923    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607927    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:17:01.612609 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607929    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607932    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607935    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607937    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607940    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607942    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607944    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607947    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607949    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607952    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607955    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607957    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607961    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607964    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607967    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607970    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607972    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607974    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607977    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.607979    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:17:01.613120 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608079    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608091    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608108    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608113    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608130    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608135    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608139    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608144    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608147    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608150    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608154    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608157    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608161    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608168    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608171    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608174    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608176    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608179    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608183    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608186    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608190    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608193    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608196    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608200    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 16:17:01.613606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608203    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608206    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608210    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608213    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608216    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608218    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608221    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608225    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608229    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608232    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608235    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608238    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608241    2579 flags.go:64] FLAG: --enable-server="true"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608244    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608249    2579 flags.go:64] FLAG: --event-burst="100"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608252    2579 flags.go:64] FLAG: --event-qps="50"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608255    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608259    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608261    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608266    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608269    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608272    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608275    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608278    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608281    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 16:17:01.614222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608284    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608287    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608290    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608293    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608296    2579 flags.go:64] FLAG: --feature-gates=""
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608299    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608302    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608305    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608309    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608312    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608315    2579 flags.go:64] FLAG: --help="false"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608317    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-154.ec2.internal"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608320    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608323    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608326    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608329    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608333    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608338    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608341    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608344    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608347    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608350    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608353    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608356    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 27 16:17:01.614938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608359    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608362    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608365    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608369    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608372    2579 flags.go:64] FLAG: --lock-file=""
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608375    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608378    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608381    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608387    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608390    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608393    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608396    2579 flags.go:64] FLAG: --logging-format="text"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608399    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608402    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608405    2579 flags.go:64] FLAG: --manifest-url=""
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608408    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608412    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608415    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608419    2579 flags.go:64] FLAG: --max-pods="110"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608422    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608425    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608428    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608430    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608433    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608436    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 16:17:01.615544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608441    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608449    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608452    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608455    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608458    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608462    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608468    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608471    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608475    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608477    2579 flags.go:64] FLAG: --port="10250"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608481    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608485    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01e97f6a654118ea3"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608488    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608491    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608494    2579 flags.go:64] FLAG: --register-node="true"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608497    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608501    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608504    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608506    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608509    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608513    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608516    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608519    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608522    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608525    2579 flags.go:64] FLAG: --runonce="false"
Jul 27 16:17:01.616220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608527    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608531    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608533    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608536    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608539    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608542    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608545    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608549    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608552    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608555    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608559    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608562    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608565    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608569    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608571    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608578    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608580    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608583    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608588    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608591    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608594    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608597    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608599    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608602    2579 flags.go:64] FLAG: --v="2"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608606    2579 flags.go:64] FLAG: --version="false"
Jul 27 16:17:01.616907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608611    2579 flags.go:64] FLAG: --vmodule=""
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608615    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.608618    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608761    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608765    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608768    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608771    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608774    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608777    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608779    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608782    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608785    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608788    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608790    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608795    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608797    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608800    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608803    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608807    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:17:01.617530 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608811    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608814    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608818    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608820    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608823    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608826    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608830    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608833    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608836    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608839    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608841    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608844    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608846    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608849    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608851    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608854    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608856    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608859    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608861    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608866    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:17:01.618028 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608869    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608872    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608875    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608877    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608880    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608882    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608885    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608889    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608891    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608894    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608897    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608899    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608902    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608904    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608907    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608910    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608912    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608915    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608919    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608921    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:17:01.618561 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608924    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608926    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608929    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608931    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608934    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608937    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608939    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608942    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608944    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608947    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608949    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608952    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608954    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608957    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608959    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608962    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608964    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608968    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608972    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:17:01.619080 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608976    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608979    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608982    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608985    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608987    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608990    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608993    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608995    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.608998    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609001    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609003    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609007    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609010    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609013    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609015    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609018    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609020    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609023    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609025    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609028    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:17:01.619536 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609031    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609033    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609036    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.609039    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.609044    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.616470    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.616493    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.616544    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.616550    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616617    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616623    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616626    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616648    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616653    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:17:01.620226 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616656    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616659    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616661    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616664    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616667    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616669    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616672    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616676    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616679    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616682    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616685    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616687    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616690    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616692    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616695    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616698    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616702    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616706    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616709    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:17:01.620597 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616712    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616715    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616717    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616720    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616723    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616727    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616730    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616733    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616736    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616738    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616741    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616744    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616746    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616748    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616751    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616753    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616756    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616758    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616761    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616764    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:17:01.621084 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616766    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616769    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616772    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616774    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616777    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616780    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616782    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616784    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616787    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616789    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616792    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616794    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616797    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616800    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616802    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616806    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616809    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616813    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616816    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616818    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:17:01.621588 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616821    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616825    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616828    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616831    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616834    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616837    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616840    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616843    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616846    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616848    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616850    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616853    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616855    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616858    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616862    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616865    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616867    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616869    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616874    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:17:01.622139 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616878    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616880    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616882    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616885    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616887    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616890    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616892    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616895    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616898    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616901    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616904    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616906    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616911    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616915    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616919    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:17:01.622652 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.616922    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.616928    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617086    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617092    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617095    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617098    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617101    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617104    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617107    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617110    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617113    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617116    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617118    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617121    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617124    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 16:17:01.623045 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617127    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617129    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617132    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617134    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617137    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617139    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617141    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617144    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617146    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617149    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617152    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617154    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617157    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617159    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617162    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617165    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617167    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617170    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617172    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 16:17:01.623419 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617175    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617177    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617180    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617183    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617185    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617188    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617190    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617193    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617195    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617197    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617201    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617205    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617208    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617211    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617213    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617216    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617219    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617223    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617226    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 16:17:01.623902 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617229    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617231    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617234    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617237    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617240    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617243    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617245    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617248    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617251    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617253    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617256    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617258    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617261    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617264    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617266    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617269    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617271    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617273    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617276    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617278    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 16:17:01.624375 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617282    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617285    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617288    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617291    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617294    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617297    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617300    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617303    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617306    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617308    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617311    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617313    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617315    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617318    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617320    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617324    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617327    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617329    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617331    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617334    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 16:17:01.625189 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617337    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617339    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617341    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617344    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617347    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617349    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617352    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:01.617354    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.617359    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.617495    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.619720    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.620942    2579 server.go:1019] "Starting client certificate rotation"
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.621056    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 16:17:01.625962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.621101    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 16:17:01.626296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.623531    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 16:17:01.626784 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.626763    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 16:17:01.633995 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.633969    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 27 16:17:01.636225 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.636197    2579 log.go:25] "Validated CRI v1 image API"
Jul 27 16:17:01.637213 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.637192    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 16:17:01.640333 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.640296    2579 fs.go:135] Filesystem UUIDs: map[5494afaa-6abf-4a92-b091-1bd8c4cf6582:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b8dc3a41-bfdb-4998-8b1d-2fad916c078c:/dev/nvme0n1p3]
Jul 27 16:17:01.640456 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.640330    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 16:17:01.645958 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.645703    2579 manager.go:216] Machine: {Timestamp:2026-07-27 16:17:01.644378397 +0000 UTC m=+0.305926402 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3112962 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28aaeae44c31ceed0313a66b30b849 SystemUUID:ec28aaea-e44c-31ce-ed03-13a66b30b849 BootID:469d7d6f-47d8-4f5c-831b-323c02ba4e4c Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0e:b6:a2:82:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0e:b6:a2:82:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:26:f5:e4:25:83:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 16:17:01.645958 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.645861    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 16:17:01.646165 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.646093    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 16:17:01.648883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.648844    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 16:17:01.649060 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.648891    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-154.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 16:17:01.650024 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.650008    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 16:17:01.650064 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.650027    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 16:17:01.650064 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.650035    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 16:17:01.650600 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.650584    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 16:17:01.651589 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.651571    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 16:17:01.654482 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.654456    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 16:17:01.654691 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.654678    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 16:17:01.655301 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655289    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 16:17:01.655351 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655311    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 16:17:01.655351 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655336    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 16:17:01.655351 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655351    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 27 16:17:01.655480 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655365    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 16:17:01.655819 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655804    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 16:17:01.655901 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.655829    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 16:17:01.658475 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.658456    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 16:17:01.660563 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.660542    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 16:17:01.660680 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.660581    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 16:17:01.662152 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662135    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662156    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662166    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662174    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662187    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662196    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662208    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662216    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 16:17:01.662249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662226    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 16:17:01.662508 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662255    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 16:17:01.662508 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662286    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 16:17:01.662508 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.662299    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 16:17:01.663033 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.663021    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 16:17:01.663081 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.663036    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 16:17:01.666517 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.666497    2579 server.go:1268] "Started kubelet"
Jul 27 16:17:01.666656 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.666604    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 16:17:01.666705 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.666600    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 16:17:01.666705 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.666694    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 16:17:01.667477 ip-10-0-129-154 systemd[1]: Started Kubernetes Kubelet.
Jul 27 16:17:01.670673 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.670626    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 16:17:01.673335 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.672759    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 16:17:01.673335 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.672882    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 16:17:01.673335 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.672909    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-154.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 16:17:01.673335 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.673046    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 16:17:01.674153 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.674131    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 16:17:01.677098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.677073    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 16:17:01.677217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.677083    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 16:17:01.677217 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.677203    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:01.677720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.677671    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 16:17:01.678042 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.677867    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 16:17:01.678042 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.677890    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 16:17:01.678194 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678135    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 16:17:01.678194 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678145    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 16:17:01.678492 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678478    2579 factory.go:153] Registering CRI-O factory
Jul 27 16:17:01.678613 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678495    2579 factory.go:223] Registration of the crio container factory successfully
Jul 27 16:17:01.678613 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678550    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 16:17:01.678613 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678560    2579 factory.go:55] Registering systemd factory
Jul 27 16:17:01.678613 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678570    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 27 16:17:01.678613 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678604    2579 factory.go:103] Registering Raw factory
Jul 27 16:17:01.678834 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.678621    2579 manager.go:1196] Started watching for new ooms in manager
Jul 27 16:17:01.679800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.679780    2579 manager.go:349] Starting recovery of all containers
Jul 27 16:17:01.680560 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.680522    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 16:17:01.680669 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.680564    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 16:17:01.681363 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.680430    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-154.ec2.internal.18c631c458760d5f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-154.ec2.internal,UID:ip-10-0-129-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-154.ec2.internal,},FirstTimestamp:2026-07-27 16:17:01.666463071 +0000 UTC m=+0.328011076,LastTimestamp:2026-07-27 16:17:01.666463071 +0000 UTC m=+0.328011076,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-154.ec2.internal,}"
Jul 27 16:17:01.691124 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.690906    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 16:17:01.692473 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.692455    2579 manager.go:354] Recovery completed
Jul 27 16:17:01.697756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.697734    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:01.699967 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.699936    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:01.700083 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.699981    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:01.700083 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.699999    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:01.700608 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.700591    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 16:17:01.700608 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.700605    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 16:17:01.700608 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.700601    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:01.700780 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.700624    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 16:17:01.702053 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.701959    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-154.ec2.internal.18c631c45a754290  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-154.ec2.internal,UID:ip-10-0-129-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-154.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-154.ec2.internal,},FirstTimestamp:2026-07-27 16:17:01.699965584 +0000 UTC m=+0.361513590,LastTimestamp:2026-07-27 16:17:01.699965584 +0000 UTC m=+0.361513590,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-154.ec2.internal,}"
Jul 27 16:17:01.703365 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.703348    2579 policy_none.go:49] "None policy: Start"
Jul 27 16:17:01.703440 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.703368    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 16:17:01.703440 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.703380    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 16:17:01.705028 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.705010    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:01.705094 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.705047    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:01.705094 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.705059    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:01.705886 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.705874    2579 policy_none.go:47] "Start"
Jul 27 16:17:01.733873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.733692    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-srp75"
Jul 27 16:17:01.745470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.745443    2579 manager.go:337] "Starting Device Plugin manager"
Jul 27 16:17:01.745703 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.745494    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 16:17:01.745703 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.745509    2579 server.go:87] "Starting device plugin registration server"
Jul 27 16:17:01.745886 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.745872    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 16:17:01.745957 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.745889    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 16:17:01.746005 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.745906    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-154.ec2.internal.18c631c45a759263  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-154.ec2.internal,UID:ip-10-0-129-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-154.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-154.ec2.internal,},FirstTimestamp:2026-07-27 16:17:01.699986019 +0000 UTC m=+0.361534026,LastTimestamp:2026-07-27 16:17:01.699986019 +0000 UTC m=+0.361534026,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-154.ec2.internal,}"
Jul 27 16:17:01.746071 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.746018    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 16:17:01.746130 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.746115    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 16:17:01.746172 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.746132    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 16:17:01.746730 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.746702    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 16:17:01.746846 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.746758    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:01.747562 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.747541    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-srp75"
Jul 27 16:17:01.846917 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.846880    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:01.858412 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.847977    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:01.858412 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.848013    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:01.858412 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.848028    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:01.858412 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.848056    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-154.ec2.internal"
Jul 27 16:17:01.882769 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.882738    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-154.ec2.internal"
Jul 27 16:17:01.882769 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.882775    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-154.ec2.internal\": node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:01.922937 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.922900    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 16:17:01.924401 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.924375    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 16:17:01.924522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.924432    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 16:17:01.924522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.924463    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 16:17:01.924522 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.924506    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 16:17:01.924689 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:01.924665    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:01.935652 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:01.935596    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:17:02.025049 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:02.025013    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:02.025049 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.025036    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"]
Jul 27 16:17:02.025211 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.025132    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:02.026854 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.026834    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:02.026979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.026866    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:02.026979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.026876    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:02.029414 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.029395    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:02.029543 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.029526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.029602 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.029569    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:02.030310 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.030291    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:02.030391 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.030324    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:02.030391 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.030335    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:02.030391 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.030292    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:02.030481 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.030395    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:02.030481 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.030405    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:02.032689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.032675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.032743 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.032704    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:02.033545 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.033527    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:02.033629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.033557    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:02.033629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.033567    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:02.055880 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:02.055851    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-154.ec2.internal\" not found" node="ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.060843 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:02.060821    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-154.ec2.internal\" not found" node="ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.080255 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.080218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14883ed3c5020af2ec5d61c737e792e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-154.ec2.internal\" (UID: \"b14883ed3c5020af2ec5d61c737e792e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.080255 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.080257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe46dd74d3722d757e67c50adb1bbd57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal\" (UID: \"fe46dd74d3722d757e67c50adb1bbd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.080401 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.080275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe46dd74d3722d757e67c50adb1bbd57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal\" (UID: \"fe46dd74d3722d757e67c50adb1bbd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.125283 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:02.125209    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:02.180740 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.180705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe46dd74d3722d757e67c50adb1bbd57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal\" (UID: \"fe46dd74d3722d757e67c50adb1bbd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.180740 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.180741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe46dd74d3722d757e67c50adb1bbd57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal\" (UID: \"fe46dd74d3722d757e67c50adb1bbd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.180877 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.180760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14883ed3c5020af2ec5d61c737e792e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-154.ec2.internal\" (UID: \"b14883ed3c5020af2ec5d61c737e792e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.180877 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.180812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe46dd74d3722d757e67c50adb1bbd57-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal\" (UID: \"fe46dd74d3722d757e67c50adb1bbd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.180877 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.180820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14883ed3c5020af2ec5d61c737e792e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-154.ec2.internal\" (UID: \"b14883ed3c5020af2ec5d61c737e792e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.180992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:02.180874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fe46dd74d3722d757e67c50adb1bbd57-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal\" (UID: \"fe46dd74d3722d757e67c50adb1bbd57\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:02.225356 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:02.225307    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.362813 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.362777    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.395021 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.394974    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:03.400913 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.400392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:03.463320 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.463223    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.563865 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.563821    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.569210 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.569182    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:17:03.656529 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.656495    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 16:17:03.656988 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.656673    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 16:17:03.656988 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.656696    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 16:17:03.664844 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.664807    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.765593 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.765499    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.776663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.776637    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 16:17:03.777902 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.777879    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 16:17:03.778063 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.777914    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 16:17:03.778063 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.777927    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeHasSufficientPID"
Jul 27 16:17:03.789730 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.789697    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:12:02 +0000 UTC" deadline="2028-05-03 11:36:08.802378292 +0000 UTC"
Jul 27 16:17:03.789730 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.789726    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15499h19m5.012655926s"
Jul 27 16:17:03.791644 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:03.791625    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:17:03.866422 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.866371    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:03.967024 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:03.966980    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:04.058377 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:04.058338    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe46dd74d3722d757e67c50adb1bbd57.slice/crio-d12476efa5c8958bd06b4ecdf861207b3bf14d6aa0f59e755996c047d9acf31d WatchSource:0}: Error finding container d12476efa5c8958bd06b4ecdf861207b3bf14d6aa0f59e755996c047d9acf31d: Status 404 returned error can't find the container with id d12476efa5c8958bd06b4ecdf861207b3bf14d6aa0f59e755996c047d9acf31d
Jul 27 16:17:04.063539 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.063502    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:17:04.067305 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.067282    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:04.069972 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:04.069947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14883ed3c5020af2ec5d61c737e792e.slice/crio-bfd9dc7d0530c96caf10c14b81f3a9066ddb54514919b482bd5df81f66471405 WatchSource:0}: Error finding container bfd9dc7d0530c96caf10c14b81f3a9066ddb54514919b482bd5df81f66471405: Status 404 returned error can't find the container with id bfd9dc7d0530c96caf10c14b81f3a9066ddb54514919b482bd5df81f66471405
Jul 27 16:17:04.167449 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.167403    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:04.268059 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.268024    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-154.ec2.internal\" not found"
Jul 27 16:17:04.294939 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.294906    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:17:04.314205 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.314140    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:04.326594 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.326566    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 16:17:04.327431 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.327415    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal"
Jul 27 16:17:04.337099 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.337073    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 16:17:04.692174 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.692083    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:17:04.693106 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.693091    2579 apiserver.go:52] "Watching apiserver"
Jul 27 16:17:04.701174 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.701155    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 16:17:04.702794 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.702770    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-vqgt2","openshift-multus/network-metrics-daemon-vzc6r","openshift-cluster-node-tuning-operator/tuned-ggttk","openshift-dns/node-resolver-klbl8","openshift-multus/multus-additional-cni-plugins-zjk58","openshift-network-diagnostics/network-check-target-m8nnm","openshift-network-operator/iptables-alerter-5jh86","openshift-ovn-kubernetes/ovnkube-node-ngl6l","kube-system/global-pull-secret-syncer-2vv29","kube-system/konnectivity-agent-r5nnt","kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww","openshift-image-registry/node-ca-btrf8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal"]
Jul 27 16:17:04.705662 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.705644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.707900 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.707876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:04.708038 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.707972    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:04.708430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.708410    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.708586 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.708567    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.708643 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.708619    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7jc59\""
Jul 27 16:17:04.709979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.709958    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.712293 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.712272    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.712601 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.712575    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5ftkm\""
Jul 27 16:17:04.712705 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.712672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.712770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.712674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 16:17:04.712770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.712717    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.714041 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.714025    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 16:17:04.714766 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.714754    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.717186 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:04.717277 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.717239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:04.717514 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717495    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.717595 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.717751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 16:17:04.717808 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 16:17:04.717897 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.717959 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.717903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bfqjr\""
Jul 27 16:17:04.718742 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.718628    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 16:17:04.718742 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.718676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nnr29\""
Jul 27 16:17:04.718742 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.718713    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.719600 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.719585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.721764 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.721745    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 16:17:04.721860 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.721775    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-v4tdq\""
Jul 27 16:17:04.722331 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.722317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.724885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.724864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.724973 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.724929    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:04.725454 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.725439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:04.725516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.725462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b2bad9d-5257-4327-a7ae-af19867afa44-tmp\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.725516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.725487    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-os-release\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.725516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.725511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-modprobe-d\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.725633 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.725539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8jq\" (UniqueName: \"kubernetes.io/projected/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-kube-api-access-xb8jq\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.726171 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-iptables-alerter-script\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.726315 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-sys\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.726315 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4564\" (UniqueName: \"kubernetes.io/projected/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-kube-api-access-p4564\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.726315 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-var-lib-kubelet\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysctl-d\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-systemd\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8v78\" (UniqueName: \"kubernetes.io/projected/1b2bad9d-5257-4327-a7ae-af19867afa44-kube-api-access-g8v78\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-system-cni-dir\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-cnibin\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfwg\" (UniqueName: \"kubernetes.io/projected/070ac775-ca28-4b0f-865d-cb36a8896da7-kube-api-access-bqfwg\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.726604 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.727231 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-tuned\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727231 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-hosts-file\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.727231 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvt4j\" (UniqueName: \"kubernetes.io/projected/ba3bcb10-939b-47af-aad5-da8823db14d1-kube-api-access-mvt4j\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:04.727231 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysctl-conf\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727231 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.726746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-lib-modules\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727476 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.727476 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysconfig\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727476 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-kubernetes\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727476 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-tmp-dir\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.727476 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-host-slash\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.727739 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727504    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-run\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727739 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-host\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.727739 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.727619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.728051 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.728032    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 16:17:04.728128 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.728090    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.728181 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.728139    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 16:17:04.728866 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.728845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 16:17:04.729098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.729079    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 16:17:04.729202 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.729145    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.729202 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.729182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.729326 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.729204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h57m8\""
Jul 27 16:17:04.731491 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.731473    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 16:17:04.732103 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.732087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.732597 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.732577    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tvrb2\""
Jul 27 16:17:04.732878 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.732864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 16:17:04.733060 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.733044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 16:17:04.734285 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.734263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 16:17:04.734511 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.734494    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.734511 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.734505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.734653 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.734542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.734653 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.734556    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5zjsh\""
Jul 27 16:17:04.736892 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.736873    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 16:17:04.737012 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.736983    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 16:17:04.737280 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.737259    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8w8np\""
Jul 27 16:17:04.737340 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.737328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 16:17:04.757757 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.757733    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wkns8"
Jul 27 16:17:04.763918 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.763897    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wkns8"
Jul 27 16:17:04.816157 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.816121    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 16:17:04.829094 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-modprobe-d\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829094 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-sys\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-socket-dir-parent\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-cni-bin\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-systemd-units\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-node-log\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovnkube-script-lib\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-sys\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09e88a58-768b-4d90-80f6-1cfc0a304f65-dbus\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829281    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-modprobe-d\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4564\" (UniqueName: \"kubernetes.io/projected/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-kube-api-access-p4564\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-var-lib-kubelet\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-conf-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-var-lib-kubelet\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8v78\" (UniqueName: \"kubernetes.io/projected/1b2bad9d-5257-4327-a7ae-af19867afa44-kube-api-access-g8v78\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-cnibin\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfwg\" (UniqueName: \"kubernetes.io/projected/070ac775-ca28-4b0f-865d-cb36a8896da7-kube-api-access-bqfwg\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-multus-certs\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-device-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.829756 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0f72c7b-bc7c-4c9c-938b-406447d310ff-cni-binary-copy\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckqgk\" (UniqueName: \"kubernetes.io/projected/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-kube-api-access-ckqgk\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-cnibin\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dde7bb49-725e-45b3-b4bb-53fff8ad3516-serviceca\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.829975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysctl-conf\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-os-release\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-daemon-config\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysctl-conf\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-log-socket\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-lib-modules\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-k8s-cni-cncf-io\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.830393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-etc-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-lib-modules\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-ovn\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-kubernetes\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-kubernetes\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-tmp-dir\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-host-slash\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-cni-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-cni-multus\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-registration-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-tmp-dir\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.830989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.831091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831095    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7lgp\" (UniqueName: \"kubernetes.io/projected/450f201b-8aba-4d30-ae94-9f5388ab02ce-kube-api-access-z7lgp\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8jq\" (UniqueName: \"kubernetes.io/projected/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-kube-api-access-xb8jq\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-iptables-alerter-script\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-netns\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-host-slash\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-kubelet\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.831503    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/070ac775-ca28-4b0f-865d-cb36a8896da7-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-etc-kubernetes\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.831619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:05.331586383 +0000 UTC m=+2.956769313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efeb8dbd-2c92-40d1-8ac2-cb53f9756db4-agent-certs\") pod \"konnectivity-agent-r5nnt\" (UID: \"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4\") " pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysctl-d\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-systemd\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-system-cni-dir\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-hostroot\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.831883 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-systemd\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-system-cni-dir\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-run-netns\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-cni-bin\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.831984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-env-overrides\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnhc\" (UniqueName: \"kubernetes.io/projected/dde7bb49-725e-45b3-b4bb-53fff8ad3516-kube-api-access-tdnhc\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p5c7\" (UniqueName: \"kubernetes.io/projected/c0f72c7b-bc7c-4c9c-938b-406447d310ff-kube-api-access-5p5c7\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-kubelet\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysctl-d\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-tuned\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-hosts-file\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvt4j\" (UniqueName: \"kubernetes.io/projected/ba3bcb10-939b-47af-aad5-da8823db14d1-kube-api-access-mvt4j\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-hosts-file\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-cnibin\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-socket-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-slash\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-var-lib-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.832637 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovnkube-config\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovn-node-metrics-cert\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-iptables-alerter-script\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efeb8dbd-2c92-40d1-8ac2-cb53f9756db4-konnectivity-ca\") pod \"konnectivity-agent-r5nnt\" (UID: \"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4\") " pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysconfig\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832439    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-run\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-sysconfig\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-host\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-cni-netd\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-sys-fs\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832586    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09e88a58-768b-4d90-80f6-1cfc0a304f65-kubelet-config\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-host\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dde7bb49-725e-45b3-b4bb-53fff8ad3516-host\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b2bad9d-5257-4327-a7ae-af19867afa44-tmp\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1b2bad9d-5257-4327-a7ae-af19867afa44-run\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-os-release\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-system-cni-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.833361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-systemd\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.833906 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.832727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/070ac775-ca28-4b0f-865d-cb36a8896da7-os-release\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.836251 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.836223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1b2bad9d-5257-4327-a7ae-af19867afa44-etc-tuned\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.836424 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.836231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b2bad9d-5257-4327-a7ae-af19867afa44-tmp\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.843814 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.843770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4564\" (UniqueName: \"kubernetes.io/projected/f6c20f89-33e0-41b5-8e2a-c1f68278cb71-kube-api-access-p4564\") pod \"iptables-alerter-5jh86\" (UID: \"f6c20f89-33e0-41b5-8e2a-c1f68278cb71\") " pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:04.847786 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.847278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8v78\" (UniqueName: \"kubernetes.io/projected/1b2bad9d-5257-4327-a7ae-af19867afa44-kube-api-access-g8v78\") pod \"tuned-ggttk\" (UID: \"1b2bad9d-5257-4327-a7ae-af19867afa44\") " pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:04.847946 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.847839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8jq\" (UniqueName: \"kubernetes.io/projected/a09ef6bc-7e7b-4803-bb17-325310a8cf7f-kube-api-access-xb8jq\") pod \"node-resolver-klbl8\" (UID: \"a09ef6bc-7e7b-4803-bb17-325310a8cf7f\") " pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:04.848638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.848608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfwg\" (UniqueName: \"kubernetes.io/projected/070ac775-ca28-4b0f-865d-cb36a8896da7-kube-api-access-bqfwg\") pod \"multus-additional-cni-plugins-zjk58\" (UID: \"070ac775-ca28-4b0f-865d-cb36a8896da7\") " pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:04.850292 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.850249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvt4j\" (UniqueName: \"kubernetes.io/projected/ba3bcb10-939b-47af-aad5-da8823db14d1-kube-api-access-mvt4j\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:04.933992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.933934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09e88a58-768b-4d90-80f6-1cfc0a304f65-kubelet-config\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.933992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.933988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dde7bb49-725e-45b3-b4bb-53fff8ad3516-host\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-system-cni-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934052    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09e88a58-768b-4d90-80f6-1cfc0a304f65-kubelet-config\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-systemd\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dde7bb49-725e-45b3-b4bb-53fff8ad3516-host\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-system-cni-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-socket-dir-parent\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-socket-dir-parent\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-systemd\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-cni-bin\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-systemd-units\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-node-log\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovnkube-script-lib\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-cni-bin\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09e88a58-768b-4d90-80f6-1cfc0a304f65-dbus\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-conf-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-node-log\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-multus-certs\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-systemd-units\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-conf-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-device-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0f72c7b-bc7c-4c9c-938b-406447d310ff-cni-binary-copy\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckqgk\" (UniqueName: \"kubernetes.io/projected/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-kube-api-access-ckqgk\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.934638 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dde7bb49-725e-45b3-b4bb-53fff8ad3516-serviceca\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-os-release\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-daemon-config\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-log-socket\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-k8s-cni-cncf-io\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-etc-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-ovn\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-cni-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-cni-multus\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-registration-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lgp\" (UniqueName: \"kubernetes.io/projected/450f201b-8aba-4d30-ae94-9f5388ab02ce-kube-api-access-z7lgp\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-netns\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-kubelet\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-etc-kubernetes\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.935379 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efeb8dbd-2c92-40d1-8ac2-cb53f9756db4-agent-certs\") pod \"konnectivity-agent-r5nnt\" (UID: \"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4\") " pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-hostroot\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-run-netns\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-cni-bin\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-env-overrides\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnhc\" (UniqueName: \"kubernetes.io/projected/dde7bb49-725e-45b3-b4bb-53fff8ad3516-kube-api-access-tdnhc\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p5c7\" (UniqueName: \"kubernetes.io/projected/c0f72c7b-bc7c-4c9c-938b-406447d310ff-kube-api-access-5p5c7\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-kubelet\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-cnibin\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-socket-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.934985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-slash\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-var-lib-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovnkube-config\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovn-node-metrics-cert\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efeb8dbd-2c92-40d1-8ac2-cb53f9756db4-konnectivity-ca\") pod \"konnectivity-agent-r5nnt\" (UID: \"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4\") " pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-cni-netd\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-sys-fs\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.936208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovnkube-script-lib\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-sys-fs\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09e88a58-768b-4d90-80f6-1cfc0a304f65-dbus\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935272    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-etc-selinux\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-netns\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-kubelet\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-etc-kubernetes\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-multus-certs\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-run-ovn-kubernetes\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-device-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.935831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-run-k8s-cni-cncf-io\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c0f72c7b-bc7c-4c9c-938b-406447d310ff-cni-binary-copy\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-etc-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-ovn\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-cni-dir\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-host-var-lib-cni-multus\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.936910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-registration-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dde7bb49-725e-45b3-b4bb-53fff8ad3516-serviceca\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.936612    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-os-release\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-hostroot\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936773    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-run-netns\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.936834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-cni-bin\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.936878    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:05.436855624 +0000 UTC m=+3.062036572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-run-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-slash\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c0f72c7b-bc7c-4c9c-938b-406447d310ff-multus-daemon-config\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-var-lib-openvswitch\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-env-overrides\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-kubelet\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-log-socket\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c0f72c7b-bc7c-4c9c-938b-406447d310ff-cnibin\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937353    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-host-cni-netd\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/450f201b-8aba-4d30-ae94-9f5388ab02ce-socket-dir\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.937797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.937766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovnkube-config\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.938665 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.938642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efeb8dbd-2c92-40d1-8ac2-cb53f9756db4-konnectivity-ca\") pod \"konnectivity-agent-r5nnt\" (UID: \"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4\") " pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.940143 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.940116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-ovn-node-metrics-cert\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.940143 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.940129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efeb8dbd-2c92-40d1-8ac2-cb53f9756db4-agent-certs\") pod \"konnectivity-agent-r5nnt\" (UID: \"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4\") " pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:04.948870 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.948840    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:04.948870 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.948874    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:04.949121 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.948888    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:04.949121 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:04.948970    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:05.448947976 +0000 UTC m=+3.074128911 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:04.951772 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.951727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckqgk\" (UniqueName: \"kubernetes.io/projected/f844d4c4-7da0-4ae1-b66e-e9f5004395e8-kube-api-access-ckqgk\") pod \"ovnkube-node-ngl6l\" (UID: \"f844d4c4-7da0-4ae1-b66e-e9f5004395e8\") " pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:04.951772 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.951748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p5c7\" (UniqueName: \"kubernetes.io/projected/c0f72c7b-bc7c-4c9c-938b-406447d310ff-kube-api-access-5p5c7\") pod \"multus-vqgt2\" (UID: \"c0f72c7b-bc7c-4c9c-938b-406447d310ff\") " pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:04.952222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.952203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7lgp\" (UniqueName: \"kubernetes.io/projected/450f201b-8aba-4d30-ae94-9f5388ab02ce-kube-api-access-z7lgp\") pod \"aws-ebs-csi-driver-node-zrwww\" (UID: \"450f201b-8aba-4d30-ae94-9f5388ab02ce\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:04.952318 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.952235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnhc\" (UniqueName: \"kubernetes.io/projected/dde7bb49-725e-45b3-b4bb-53fff8ad3516-kube-api-access-tdnhc\") pod \"node-ca-btrf8\" (UID: \"dde7bb49-725e-45b3-b4bb-53fff8ad3516\") " pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:04.965689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.965632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal" event={"ID":"b14883ed3c5020af2ec5d61c737e792e","Type":"ContainerStarted","Data":"bfd9dc7d0530c96caf10c14b81f3a9066ddb54514919b482bd5df81f66471405"}
Jul 27 16:17:04.967449 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:04.967415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal" event={"ID":"fe46dd74d3722d757e67c50adb1bbd57","Type":"ContainerStarted","Data":"d12476efa5c8958bd06b4ecdf861207b3bf14d6aa0f59e755996c047d9acf31d"}
Jul 27 16:17:05.016363 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.016326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klbl8"
Jul 27 16:17:05.022221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.022191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5jh86"
Jul 27 16:17:05.030726 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.030607    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09ef6bc_7e7b_4803_bb17_325310a8cf7f.slice/crio-7274726378ce20a8c11fc8357076bd9991aac1d8f6021e500e1396f234db8b89 WatchSource:0}: Error finding container 7274726378ce20a8c11fc8357076bd9991aac1d8f6021e500e1396f234db8b89: Status 404 returned error can't find the container with id 7274726378ce20a8c11fc8357076bd9991aac1d8f6021e500e1396f234db8b89
Jul 27 16:17:05.032746 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.032725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggttk"
Jul 27 16:17:05.035404 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.035143    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c20f89_33e0_41b5_8e2a_c1f68278cb71.slice/crio-b8827679f3112a379ba43984c532d1b7527f7bf647b9cf08f747e06d105b51c5 WatchSource:0}: Error finding container b8827679f3112a379ba43984c532d1b7527f7bf647b9cf08f747e06d105b51c5: Status 404 returned error can't find the container with id b8827679f3112a379ba43984c532d1b7527f7bf647b9cf08f747e06d105b51c5
Jul 27 16:17:05.038312 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.038289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjk58"
Jul 27 16:17:05.045549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.045517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww"
Jul 27 16:17:05.050330 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.050297    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2bad9d_5257_4327_a7ae_af19867afa44.slice/crio-0357ba7863bdab10f45f2b27290df66924b2866a0696dd2f75a55000e28e1bca WatchSource:0}: Error finding container 0357ba7863bdab10f45f2b27290df66924b2866a0696dd2f75a55000e28e1bca: Status 404 returned error can't find the container with id 0357ba7863bdab10f45f2b27290df66924b2866a0696dd2f75a55000e28e1bca
Jul 27 16:17:05.050448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.050396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vqgt2"
Jul 27 16:17:05.054392 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.054358    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070ac775_ca28_4b0f_865d_cb36a8896da7.slice/crio-fe44f5a9ab6a214eb71d9473c6f575fd6039a7915692b24b33906010a46f1cba WatchSource:0}: Error finding container fe44f5a9ab6a214eb71d9473c6f575fd6039a7915692b24b33906010a46f1cba: Status 404 returned error can't find the container with id fe44f5a9ab6a214eb71d9473c6f575fd6039a7915692b24b33906010a46f1cba
Jul 27 16:17:05.057057 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.056995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:05.060354 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.060272    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450f201b_8aba_4d30_ae94_9f5388ab02ce.slice/crio-d816798ecd85744e424379dfc18526212832ba1f24c4e86019ba136101219fe5 WatchSource:0}: Error finding container d816798ecd85744e424379dfc18526212832ba1f24c4e86019ba136101219fe5: Status 404 returned error can't find the container with id d816798ecd85744e424379dfc18526212832ba1f24c4e86019ba136101219fe5
Jul 27 16:17:05.062341 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.062314    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:05.066307 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.066155    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-btrf8"
Jul 27 16:17:05.070256 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.070227    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0f72c7b_bc7c_4c9c_938b_406447d310ff.slice/crio-30a33f6cd85f555a16b7c1228086c6788e9e28510c8518b5c875a2099dc66942 WatchSource:0}: Error finding container 30a33f6cd85f555a16b7c1228086c6788e9e28510c8518b5c875a2099dc66942: Status 404 returned error can't find the container with id 30a33f6cd85f555a16b7c1228086c6788e9e28510c8518b5c875a2099dc66942
Jul 27 16:17:05.075357 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.075318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf844d4c4_7da0_4ae1_b66e_e9f5004395e8.slice/crio-3ee7a005d14770c6d557ccb66d46587a3b37970c2d858bfd17db3915a06036ec WatchSource:0}: Error finding container 3ee7a005d14770c6d557ccb66d46587a3b37970c2d858bfd17db3915a06036ec: Status 404 returned error can't find the container with id 3ee7a005d14770c6d557ccb66d46587a3b37970c2d858bfd17db3915a06036ec
Jul 27 16:17:05.084618 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.084583    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefeb8dbd_2c92_40d1_8ac2_cb53f9756db4.slice/crio-9934c88876cdddde504672ad4f3ac2cb4930a6de6923d0cc35261a181958ae11 WatchSource:0}: Error finding container 9934c88876cdddde504672ad4f3ac2cb4930a6de6923d0cc35261a181958ae11: Status 404 returned error can't find the container with id 9934c88876cdddde504672ad4f3ac2cb4930a6de6923d0cc35261a181958ae11
Jul 27 16:17:05.087778 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:05.087707    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde7bb49_725e_45b3_b4bb_53fff8ad3516.slice/crio-0cc5aa10397448b52ebeeebb862a9386efe55bdb6b0033a307d9138177e0498c WatchSource:0}: Error finding container 0cc5aa10397448b52ebeeebb862a9386efe55bdb6b0033a307d9138177e0498c: Status 404 returned error can't find the container with id 0cc5aa10397448b52ebeeebb862a9386efe55bdb6b0033a307d9138177e0498c
Jul 27 16:17:05.337758 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.337659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:05.337926 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.337807    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:05.337926 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.337868    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:06.337852084 +0000 UTC m=+3.963033015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:05.438811 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.438764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:05.438979 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.438941    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:05.439068 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.439032    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:06.439016505 +0000 UTC m=+4.064197448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:05.539854 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.539808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:05.540083 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.540022    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:05.540083 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.540045    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:05.540083 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.540056    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:05.540232 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.540116    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:06.540096553 +0000 UTC m=+4.165277484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:05.767815 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.767758    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:12:04 +0000 UTC" deadline="2028-04-23 01:15:46.72884439 +0000 UTC"
Jul 27 16:17:05.767815 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.767805    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15248h58m40.961044192s"
Jul 27 16:17:05.960194 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.959873    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 16:17:05.962174 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.961404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:05.962174 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.961543    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:05.962174 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.961937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:05.962174 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:05.962043    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:05.971781 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.971697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-btrf8" event={"ID":"dde7bb49-725e-45b3-b4bb-53fff8ad3516","Type":"ContainerStarted","Data":"0cc5aa10397448b52ebeeebb862a9386efe55bdb6b0033a307d9138177e0498c"}
Jul 27 16:17:05.973437 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.973225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggttk" event={"ID":"1b2bad9d-5257-4327-a7ae-af19867afa44","Type":"ContainerStarted","Data":"0357ba7863bdab10f45f2b27290df66924b2866a0696dd2f75a55000e28e1bca"}
Jul 27 16:17:05.974802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.974723    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5jh86" event={"ID":"f6c20f89-33e0-41b5-8e2a-c1f68278cb71","Type":"ContainerStarted","Data":"b8827679f3112a379ba43984c532d1b7527f7bf647b9cf08f747e06d105b51c5"}
Jul 27 16:17:05.976246 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.976218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klbl8" event={"ID":"a09ef6bc-7e7b-4803-bb17-325310a8cf7f","Type":"ContainerStarted","Data":"7274726378ce20a8c11fc8357076bd9991aac1d8f6021e500e1396f234db8b89"}
Jul 27 16:17:05.978245 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.978217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5nnt" event={"ID":"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4","Type":"ContainerStarted","Data":"9934c88876cdddde504672ad4f3ac2cb4930a6de6923d0cc35261a181958ae11"}
Jul 27 16:17:05.980459 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.980431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"3ee7a005d14770c6d557ccb66d46587a3b37970c2d858bfd17db3915a06036ec"}
Jul 27 16:17:05.984150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.984114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqgt2" event={"ID":"c0f72c7b-bc7c-4c9c-938b-406447d310ff","Type":"ContainerStarted","Data":"30a33f6cd85f555a16b7c1228086c6788e9e28510c8518b5c875a2099dc66942"}
Jul 27 16:17:05.986438 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.986406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww" event={"ID":"450f201b-8aba-4d30-ae94-9f5388ab02ce","Type":"ContainerStarted","Data":"d816798ecd85744e424379dfc18526212832ba1f24c4e86019ba136101219fe5"}
Jul 27 16:17:05.987832 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:05.987805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerStarted","Data":"fe44f5a9ab6a214eb71d9473c6f575fd6039a7915692b24b33906010a46f1cba"}
Jul 27 16:17:06.347983 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.347951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:06.348179 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.348137    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:06.348247 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.348202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:08.348183868 +0000 UTC m=+5.973364801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:06.452778 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.452730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:06.452944 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.452900    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:06.453053 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.452971    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:08.45295036 +0000 UTC m=+6.078131316 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:06.553650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.553322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:06.553851 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.553737    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:06.553851 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.553763    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:06.553851 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.553776    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:06.553851 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.553835    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:08.553812529 +0000 UTC m=+6.178993464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:06.768540 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.768460    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:12:04 +0000 UTC" deadline="2027-12-21 14:19:31.446330639 +0000 UTC"
Jul 27 16:17:06.768540 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.768537    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12286h2m24.677798737s"
Jul 27 16:17:06.962250 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.962220    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:06.962464 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:06.962354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:06.994109 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:06.994066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal" event={"ID":"b14883ed3c5020af2ec5d61c737e792e","Type":"ContainerStarted","Data":"79e24c2d04d64887a3665f024047bd5379eff21a8ec5fe6bcd17ed1f4a6f4acc"}
Jul 27 16:17:07.961770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:07.961731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:07.962282 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:07.961880    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:07.962368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:07.962293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:07.962431 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:07.962386    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:08.369871 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:08.369785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:08.370056 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.369970    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:08.370119 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.370061    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:12.370039475 +0000 UTC m=+9.995220409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:08.470530 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:08.470488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:08.470734 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.470661    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:08.470872 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.470850    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:12.470741689 +0000 UTC m=+10.095922627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:08.574358 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:08.574313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:08.574600 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.574576    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:08.574689 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.574607    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:08.574689 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.574621    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:08.574689 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.574684    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:12.574663967 +0000 UTC m=+10.199844899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:08.969076 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:08.969044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:08.969563 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:08.969230    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:09.961848 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:09.961805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:09.961848 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:09.961837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:09.962131 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:09.961936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:09.962131 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:09.962069    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:10.962494 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:10.961864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:10.962494 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:10.961997    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:11.961388 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:11.961340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:11.961586 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:11.961398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:11.961586 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:11.961495    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:11.961717 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:11.961611    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:12.410335 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:12.410238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:12.410749 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.410405    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:12.410749 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.410463    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:20.410449956 +0000 UTC m=+18.035630886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:12.511728 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:12.511679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:12.511907 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.511889    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:12.511983 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.511963    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:20.511943241 +0000 UTC m=+18.137124195 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:12.613133 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:12.613093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:12.613314 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.613290    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:12.613369 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.613323    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:12.613369 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.613338    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:12.613457 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.613407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:20.613385895 +0000 UTC m=+18.238566831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:12.962495 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:12.962459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:12.962714 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:12.962561    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:13.961811 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:13.961770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:13.962446 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:13.961907    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:13.962446 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:13.962339    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:13.962446 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:13.962433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:14.961157 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:14.961109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:14.961358 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:14.961290    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:15.961650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:15.961611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:15.962167 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:15.961661    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:15.962167 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:15.961754    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:15.962167 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:15.961881    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:16.961129 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:16.961092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:16.961311 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:16.961239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:17.962128 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:17.962082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:17.962623 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:17.962082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:17.962623 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:17.962229    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:17.962623 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:17.962367    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:18.961348 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:18.961310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:18.961569 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:18.961452    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:19.961677 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:19.961638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:19.961677 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:19.961666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:19.962208 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:19.961775    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:19.962208 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:19.961886    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:20.471976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:20.471930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:20.472231 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.472073    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:20.472231 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.472139    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:36.472120623 +0000 UTC m=+34.097301555 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:20.572954 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:20.572918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:20.573170 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.573104    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:20.573232 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.573189    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:36.573167663 +0000 UTC m=+34.198348594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:20.673633 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:20.673526    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:20.673799 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.673684    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:20.673799 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.673706    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:20.673799 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.673717    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:20.673799 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.673776    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:17:36.673758199 +0000 UTC m=+34.298939146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:20.962097 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:20.962061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:20.962598 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:20.962227    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:21.961357 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:21.961315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:21.961357 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:21.961346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:21.961666 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:21.961471    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:21.962059 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:21.961994    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:22.962492 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:22.962453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:22.962980 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:22.962581    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:23.961932 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:23.961881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:23.962131 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:23.961902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:23.962131 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:23.962038    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:23.962131 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:23.962106    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:24.962287 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:24.962085    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:24.963160 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:24.962401    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:25.038385 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.037892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggttk" event={"ID":"1b2bad9d-5257-4327-a7ae-af19867afa44","Type":"ContainerStarted","Data":"447dc65d7e9d2b8939c56216e6f73ee81ddb3799562bea500a44889ba24411d5"}
Jul 27 16:17:25.039727 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.039692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klbl8" event={"ID":"a09ef6bc-7e7b-4803-bb17-325310a8cf7f","Type":"ContainerStarted","Data":"38b04107c94d2e0e5e88b4378d9052471d4d6a9ec0df6e2d058996e80ee20f81"}
Jul 27 16:17:25.041591 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.041562    2579 generic.go:356] "Generic (PLEG): container finished" podID="fe46dd74d3722d757e67c50adb1bbd57" containerID="465722c9936df1b609f876f93101e5ca101e146f125e45686e0159f64aea3d46" exitCode=0
Jul 27 16:17:25.041749 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.041656    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal" event={"ID":"fe46dd74d3722d757e67c50adb1bbd57","Type":"ContainerDied","Data":"465722c9936df1b609f876f93101e5ca101e146f125e45686e0159f64aea3d46"}
Jul 27 16:17:25.043539 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.043509    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r5nnt" event={"ID":"efeb8dbd-2c92-40d1-8ac2-cb53f9756db4","Type":"ContainerStarted","Data":"02490c31207628788dca8f2168c827889e5f0e10d4548604a44c55721b74c84c"}
Jul 27 16:17:25.047391 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.047349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"63d6b56abd9a9b8f33070bdf28132c8e349a95af2368923da315cf0f8b835de9"}
Jul 27 16:17:25.047524 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.047451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"2c954b2e3ca7b46d87e42934b4104efd713bfc0bf1b974b527260b6e5e5d69ae"}
Jul 27 16:17:25.047524 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.047519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"068abd0ba29967c63b41d460e524b3ddf0fe982df58d69ddc965276985b315d8"}
Jul 27 16:17:25.047644 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.047581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"c9b727444097adab21b343fd6dee62c407c6dc24635defec1e11b0ad81f4a8aa"}
Jul 27 16:17:25.047644 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.047626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"3f1c167eb50d50e3a9da3f6f2a22e384341abb91330fa416b2eb520d1ac0eaa6"}
Jul 27 16:17:25.047736 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.047673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"0e45708bd67acdfdc01ee60a7b42ebfdbde7b41a7399a0487d1badc43329a2d8"}
Jul 27 16:17:25.049458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.049249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vqgt2" event={"ID":"c0f72c7b-bc7c-4c9c-938b-406447d310ff","Type":"ContainerStarted","Data":"f8153a7ab8520accf08541d0ad4faebf21d91b93a185c60c72619f33a66a4ed8"}
Jul 27 16:17:25.054139 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.054102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww" event={"ID":"450f201b-8aba-4d30-ae94-9f5388ab02ce","Type":"ContainerStarted","Data":"70b29a302733251861e49d68609f162b570a2bc0bc081a006888116e8aacb824"}
Jul 27 16:17:25.056105 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.056068    2579 generic.go:356] "Generic (PLEG): container finished" podID="070ac775-ca28-4b0f-865d-cb36a8896da7" containerID="d7d703037e6a9eceb9de61ddabcc363b6ca43ae0c3a0203a74a6596b7d0fa0bb" exitCode=0
Jul 27 16:17:25.056234 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.056156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerDied","Data":"d7d703037e6a9eceb9de61ddabcc363b6ca43ae0c3a0203a74a6596b7d0fa0bb"}
Jul 27 16:17:25.056234 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.056158    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ggttk" podStartSLOduration=2.807951688 podStartE2EDuration="22.056142491s" totalImagesPullingTime="19.248190803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:25.05574768 +0000 UTC m=+22.680928633" watchObservedRunningTime="2026-07-27 16:17:25.056142491 +0000 UTC m=+22.681323441"
Jul 27 16:17:25.056723 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.056690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-154.ec2.internal" podStartSLOduration=21.056681232 podStartE2EDuration="21.056681232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:07.016310744 +0000 UTC m=+4.641491698" watchObservedRunningTime="2026-07-27 16:17:25.056681232 +0000 UTC m=+22.681862236"
Jul 27 16:17:25.061391 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.061354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-btrf8" event={"ID":"dde7bb49-725e-45b3-b4bb-53fff8ad3516","Type":"ContainerStarted","Data":"3f6ff98de489b73e3f80bfc706126c1693778b8fb935ff1c1e2d0ef2b0f5165f"}
Jul 27 16:17:25.092808 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.092750    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vqgt2" podStartSLOduration=2.863501462 podStartE2EDuration="22.092733864s" totalImagesPullingTime="19.229232402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:25.09223971 +0000 UTC m=+22.717420665" watchObservedRunningTime="2026-07-27 16:17:25.092733864 +0000 UTC m=+22.717914816"
Jul 27 16:17:25.093023 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.092960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-klbl8" podStartSLOduration=14.841308061 podStartE2EDuration="23.092952905s" totalImagesPullingTime="8.251644844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:25.072778046 +0000 UTC m=+22.697959003" watchObservedRunningTime="2026-07-27 16:17:25.092952905 +0000 UTC m=+22.718133861"
Jul 27 16:17:25.107869 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.107818    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r5nnt" podStartSLOduration=13.916526983 podStartE2EDuration="22.107802252s" totalImagesPullingTime="8.191275269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:25.107559348 +0000 UTC m=+22.732740301" watchObservedRunningTime="2026-07-27 16:17:25.107802252 +0000 UTC m=+22.732983205"
Jul 27 16:17:25.165169 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.165111    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-btrf8" podStartSLOduration=13.977752249 podStartE2EDuration="22.165091855s" totalImagesPullingTime="8.187339606s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:25.16450575 +0000 UTC m=+22.789686716" watchObservedRunningTime="2026-07-27 16:17:25.165091855 +0000 UTC m=+22.790272879"
Jul 27 16:17:25.961710 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.961491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:25.961913 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:25.961491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:25.961913 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:25.961825    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:25.961913 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:25.961873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:26.065605 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:26.065567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5jh86" event={"ID":"f6c20f89-33e0-41b5-8e2a-c1f68278cb71","Type":"ContainerStarted","Data":"7f66137c2142f5d9c17f54ade1d7dcffbdea9c766e975f6f1f5cd04ddc445498"}
Jul 27 16:17:26.068042 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:26.067989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal" event={"ID":"fe46dd74d3722d757e67c50adb1bbd57","Type":"ContainerStarted","Data":"41e1205ad0d79cbad237103b47ce7cc6125ca70534ab9f5321b3ad42ee307ae5"}
Jul 27 16:17:26.083161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:26.083100    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5jh86" podStartSLOduration=14.840825722 podStartE2EDuration="23.083075126s" totalImagesPullingTime="8.242249404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:26.0825351 +0000 UTC m=+23.707716055" watchObservedRunningTime="2026-07-27 16:17:26.083075126 +0000 UTC m=+23.708256268"
Jul 27 16:17:26.099600 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:26.099542    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-154.ec2.internal" podStartSLOduration=22.099520363 podStartE2EDuration="22.099520363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:26.09943967 +0000 UTC m=+23.724620624" watchObservedRunningTime="2026-07-27 16:17:26.099520363 +0000 UTC m=+23.724701390"
Jul 27 16:17:26.961250 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:26.961209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:26.961513 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:26.961363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:27.961103 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:27.961058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:27.961561 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:27.961117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:27.961561 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:27.961214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:27.961561 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:27.961345    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:28.077271 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:28.077207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"3c5ff7ff632138e7f484ccc55be97243d9aefc380b2066610243eb12a4be1ba5"}
Jul 27 16:17:28.226454 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:28.226359    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:28.227058 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:28.227032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:28.961927 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:28.961875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:28.962392 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:28.962061    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:29.962289 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:29.962083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:29.962700 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:29.962130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:29.962700 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:29.962393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:29.962700 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:29.962472    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:30.086747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.086631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" event={"ID":"f844d4c4-7da0-4ae1-b66e-e9f5004395e8","Type":"ContainerStarted","Data":"c0cd2cb49c788455cb688a54639ab841a479f51b70fd667ed96621d2a310470d"}
Jul 27 16:17:30.087376 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.087351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:30.087534 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.087514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:30.087614 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.087603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:30.089844 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.089810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerStarted","Data":"3cc32395a65609c3cfe8462c16d70c1a85d81ea3e05eaf1795780a97dcaaa927"}
Jul 27 16:17:30.104747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.104692    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:30.104910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.104894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:17:30.118942 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.118895    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l" podStartSLOduration=7.776996593 podStartE2EDuration="27.1188803s" totalImagesPullingTime="19.341883707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:30.11844981 +0000 UTC m=+27.743630762" watchObservedRunningTime="2026-07-27 16:17:30.1188803 +0000 UTC m=+27.744061252"
Jul 27 16:17:30.509899 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.509872    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 16:17:30.804943 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.804786    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T16:17:30.509891454Z","UUID":"cc0aeec8-8663-4747-9d8b-f73a97ba7660","Handler":null,"Name":"","Endpoint":""}
Jul 27 16:17:30.806543 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.806511    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 16:17:30.806715 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.806556    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 16:17:30.961308 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:30.961268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:30.961528 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:30.961430    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:31.094101 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.094051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww" event={"ID":"450f201b-8aba-4d30-ae94-9f5388ab02ce","Type":"ContainerStarted","Data":"2c596ab668307429fc468c9f1ebf332860eff255b9da46ccc9702a29c11a8e59"}
Jul 27 16:17:31.096021 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.095973    2579 generic.go:356] "Generic (PLEG): container finished" podID="070ac775-ca28-4b0f-865d-cb36a8896da7" containerID="3cc32395a65609c3cfe8462c16d70c1a85d81ea3e05eaf1795780a97dcaaa927" exitCode=0
Jul 27 16:17:31.096163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.096032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerDied","Data":"3cc32395a65609c3cfe8462c16d70c1a85d81ea3e05eaf1795780a97dcaaa927"}
Jul 27 16:17:31.517552 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.517350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2vv29"]
Jul 27 16:17:31.517741 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.517680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:31.517794 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:31.517774    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:31.520666 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.520625    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vzc6r"]
Jul 27 16:17:31.520838 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.520804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:31.520969 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:31.520944    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:31.521291 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.521265    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m8nnm"]
Jul 27 16:17:31.521427 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:31.521379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:31.521500 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:31.521479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:32.964934 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:32.964902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:32.965417 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:32.964902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:32.965417 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:32.965029    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:32.965417 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:32.964902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:32.965417 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:32.965089    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:32.965417 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:32.965154    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:33.102272 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:33.102236    2579 generic.go:356] "Generic (PLEG): container finished" podID="070ac775-ca28-4b0f-865d-cb36a8896da7" containerID="5c3959c8f63459f35c104eb203c3fa5387c5c076eafeea2a8e8de50e85c7f20e" exitCode=0
Jul 27 16:17:33.102463 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:33.102287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerDied","Data":"5c3959c8f63459f35c104eb203c3fa5387c5c076eafeea2a8e8de50e85c7f20e"}
Jul 27 16:17:34.160548 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:34.160517    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:34.161150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:34.160689    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 16:17:34.161459 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:34.161438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r5nnt"
Jul 27 16:17:34.961950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:34.961903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:34.962158 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:34.961910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:34.962158 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:34.962049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:34.962158 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:34.962052    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:34.962158 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:34.962139    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:34.962374 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:34.962214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:35.109568 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:35.109530    2579 generic.go:356] "Generic (PLEG): container finished" podID="070ac775-ca28-4b0f-865d-cb36a8896da7" containerID="781d96989043dc1ecf8f4f3f7daed9f6ca993ab8523aae721c2fb61cb9a2f754" exitCode=0
Jul 27 16:17:35.109726 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:35.109601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerDied","Data":"781d96989043dc1ecf8f4f3f7daed9f6ca993ab8523aae721c2fb61cb9a2f754"}
Jul 27 16:17:36.115111 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.114798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww" event={"ID":"450f201b-8aba-4d30-ae94-9f5388ab02ce","Type":"ContainerStarted","Data":"d1890fac6cc9d25a317e33156cc164d95f892f6af1e7a93abefb7e90b116ac33"}
Jul 27 16:17:36.136446 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.136382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zrwww" podStartSLOduration=14.647965679 podStartE2EDuration="33.136361262s" totalImagesPullingTime="18.488395583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:36.13595916 +0000 UTC m=+33.761140124" watchObservedRunningTime="2026-07-27 16:17:36.136361262 +0000 UTC m=+33.761542220"
Jul 27 16:17:36.497095 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.497055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:36.497306 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.497243    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:36.497361 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.497333    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:08.497312146 +0000 UTC m=+66.122493083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 16:17:36.597804 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.597763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:36.598045 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.597924    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:36.598045 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.597987    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret podName:09e88a58-768b-4d90-80f6-1cfc0a304f65 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:08.597972055 +0000 UTC m=+66.223152989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret") pod "global-pull-secret-syncer-2vv29" (UID: "09e88a58-768b-4d90-80f6-1cfc0a304f65") : object "kube-system"/"original-pull-secret" not registered
Jul 27 16:17:36.698395 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.698351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:36.698592 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.698505    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 16:17:36.698592 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.698520    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 16:17:36.698592 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.698530    2579 projected.go:196] Error preparing data for projected volume kube-api-access-cbqrm for pod openshift-network-diagnostics/network-check-target-m8nnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:36.698592 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.698591    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm podName:1124f4b6-09d2-4b8b-9c50-46a34e10f6b8 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:08.698570123 +0000 UTC m=+66.323751054 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cbqrm" (UniqueName: "kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm") pod "network-check-target-m8nnm" (UID: "1124f4b6-09d2-4b8b-9c50-46a34e10f6b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 16:17:36.962108 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.962068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:36.962108 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.962100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:36.962357 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:36.962100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:36.962357 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.962215    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m8nnm" podUID="1124f4b6-09d2-4b8b-9c50-46a34e10f6b8"
Jul 27 16:17:36.962357 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.962297    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vzc6r" podUID="ba3bcb10-939b-47af-aad5-da8823db14d1"
Jul 27 16:17:36.962480 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:36.962374    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vv29" podUID="09e88a58-768b-4d90-80f6-1cfc0a304f65"
Jul 27 16:17:37.223225 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.223194    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-154.ec2.internal" event="NodeReady"
Jul 27 16:17:37.223819 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.223363    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 16:17:37.270338 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.270262    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pr7qc"]
Jul 27 16:17:37.284430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.284383    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tx9vb"]
Jul 27 16:17:37.284657 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.284549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.287689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.287379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2mnq\""
Jul 27 16:17:37.287689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.287425    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 16:17:37.287689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.287430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 16:17:37.298217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.298188    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pr7qc"]
Jul 27 16:17:37.298368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.298239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tx9vb"]
Jul 27 16:17:37.298368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.298313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.301620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.301423    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 16:17:37.301620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.301440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9rtzs\""
Jul 27 16:17:37.301620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.301475    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 16:17:37.301620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.301423    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 16:17:37.403882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.403836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vwjv\" (UniqueName: \"kubernetes.io/projected/799b4fdb-c34f-434f-977d-323105f3709b-kube-api-access-6vwjv\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.404112 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.403933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/799b4fdb-c34f-434f-977d-323105f3709b-config-volume\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.404112 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.403960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgzn\" (UniqueName: \"kubernetes.io/projected/5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7-kube-api-access-spgzn\") pod \"ingress-canary-tx9vb\" (UID: \"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7\") " pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.404112 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.404020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7-cert\") pod \"ingress-canary-tx9vb\" (UID: \"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7\") " pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.404324 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.404161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.404324 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.404198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799b4fdb-c34f-434f-977d-323105f3709b-tmp-dir\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.505271 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7-cert\") pod \"ingress-canary-tx9vb\" (UID: \"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7\") " pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.505271 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.505537 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799b4fdb-c34f-434f-977d-323105f3709b-tmp-dir\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.505537 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vwjv\" (UniqueName: \"kubernetes.io/projected/799b4fdb-c34f-434f-977d-323105f3709b-kube-api-access-6vwjv\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.505537 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/799b4fdb-c34f-434f-977d-323105f3709b-config-volume\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.505537 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:37.505389    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:37.505537 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgzn\" (UniqueName: \"kubernetes.io/projected/5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7-kube-api-access-spgzn\") pod \"ingress-canary-tx9vb\" (UID: \"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7\") " pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.505537 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:37.505476    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:38.005452723 +0000 UTC m=+35.630633676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:37.505837 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.505698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799b4fdb-c34f-434f-977d-323105f3709b-tmp-dir\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.506206 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.506189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/799b4fdb-c34f-434f-977d-323105f3709b-config-volume\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.511145 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.511115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7-cert\") pod \"ingress-canary-tx9vb\" (UID: \"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7\") " pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.514864 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.514834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vwjv\" (UniqueName: \"kubernetes.io/projected/799b4fdb-c34f-434f-977d-323105f3709b-kube-api-access-6vwjv\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:37.515068 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.515018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgzn\" (UniqueName: \"kubernetes.io/projected/5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7-kube-api-access-spgzn\") pod \"ingress-canary-tx9vb\" (UID: \"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7\") " pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.610234 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.610196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tx9vb"
Jul 27 16:17:37.816084 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:37.815748    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tx9vb"]
Jul 27 16:17:38.010712 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.010673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:38.010962 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:38.010850    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:38.010962 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:38.010935    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:39.010916168 +0000 UTC m=+36.636097098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:38.961862 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.961821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:17:38.962895 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.961828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:17:38.962895 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.961986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:17:38.965118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.964902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 16:17:38.966057 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.966028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 16:17:38.966057 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.966046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 16:17:38.966255 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.966034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9d9gv\""
Jul 27 16:17:38.966255 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.966051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fft22\""
Jul 27 16:17:38.966255 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:38.966034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 16:17:39.019231 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:39.019182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:39.019420 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:39.019366    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:39.019472 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:39.019442    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:41.019420844 +0000 UTC m=+38.644601775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:39.277199 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:17:39.277100    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4bb02e_b6a1_433f_9a49_f1e5882aa4e7.slice/crio-3c02ec32a790753fa218a7d5ea4b2a618ee6c8f0757112cc068926ea1c18e9cf WatchSource:0}: Error finding container 3c02ec32a790753fa218a7d5ea4b2a618ee6c8f0757112cc068926ea1c18e9cf: Status 404 returned error can't find the container with id 3c02ec32a790753fa218a7d5ea4b2a618ee6c8f0757112cc068926ea1c18e9cf
Jul 27 16:17:40.129183 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:40.129140    2579 generic.go:356] "Generic (PLEG): container finished" podID="070ac775-ca28-4b0f-865d-cb36a8896da7" containerID="c1c349acbbf848ecdc061d5a25d3228e790793851bf1e7d1f68705f7dd2b712e" exitCode=0
Jul 27 16:17:40.129702 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:40.129225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerDied","Data":"c1c349acbbf848ecdc061d5a25d3228e790793851bf1e7d1f68705f7dd2b712e"}
Jul 27 16:17:40.131082 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:40.130996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tx9vb" event={"ID":"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7","Type":"ContainerStarted","Data":"3c02ec32a790753fa218a7d5ea4b2a618ee6c8f0757112cc068926ea1c18e9cf"}
Jul 27 16:17:41.033938 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:41.033893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:41.034141 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:41.034094    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:41.034184 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:41.034175    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:45.034153326 +0000 UTC m=+42.659334260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:41.138157 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:41.138117    2579 generic.go:356] "Generic (PLEG): container finished" podID="070ac775-ca28-4b0f-865d-cb36a8896da7" containerID="2f58fc65b3dd7e70fe9b804609a93c2cd26eb0796043d8293631525a46b37665" exitCode=0
Jul 27 16:17:41.138606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:41.138169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerDied","Data":"2f58fc65b3dd7e70fe9b804609a93c2cd26eb0796043d8293631525a46b37665"}
Jul 27 16:17:42.143961 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:42.143922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjk58" event={"ID":"070ac775-ca28-4b0f-865d-cb36a8896da7","Type":"ContainerStarted","Data":"723d3a4a58ce4a05c9b4ce5807a70c05f8e579edb456dc4c5778dd2bc7b76c4c"}
Jul 27 16:17:42.171798 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:42.171592    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjk58" podStartSLOduration=8.303284275 podStartE2EDuration="39.171572701s" totalImagesPullingTime="30.868288426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:42.170256703 +0000 UTC m=+39.795437668" watchObservedRunningTime="2026-07-27 16:17:42.171572701 +0000 UTC m=+39.796753654"
Jul 27 16:17:43.147528 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:43.147484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tx9vb" event={"ID":"5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7","Type":"ContainerStarted","Data":"78c203252bc883647583357fadff908c460858a9f73d33e50591ff5e908318a2"}
Jul 27 16:17:43.166692 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:43.166639    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tx9vb" podStartSLOduration=3.197056491 podStartE2EDuration="6.166620425s" totalImagesPullingTime="2.969563934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:17:43.16587969 +0000 UTC m=+40.791060723" watchObservedRunningTime="2026-07-27 16:17:43.166620425 +0000 UTC m=+40.791801378"
Jul 27 16:17:45.061440 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:45.061404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:45.061818 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:45.061565    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:45.061818 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:45.061636    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:17:53.061617425 +0000 UTC m=+50.686798356 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:17:53.121387 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:17:53.121327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:17:53.121821 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:53.121487    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:17:53.121821 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:17:53.121554    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:18:09.121537909 +0000 UTC m=+66.746718840 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:18:02.118796 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:02.118765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ngl6l"
Jul 27 16:18:08.530767 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.530720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:18:08.533686 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.533660    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 16:18:08.541915 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:08.541890    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 16:18:08.541986 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:08.541966    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs podName:ba3bcb10-939b-47af-aad5-da8823db14d1 nodeName:}" failed. No retries permitted until 2026-07-27 16:19:12.541946676 +0000 UTC m=+130.167127608 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs") pod "network-metrics-daemon-vzc6r" (UID: "ba3bcb10-939b-47af-aad5-da8823db14d1") : secret "metrics-daemon-secret" not found
Jul 27 16:18:08.631745 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.631694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:18:08.634769 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.634749    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 16:18:08.646059 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.646023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09e88a58-768b-4d90-80f6-1cfc0a304f65-original-pull-secret\") pod \"global-pull-secret-syncer-2vv29\" (UID: \"09e88a58-768b-4d90-80f6-1cfc0a304f65\") " pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:18:08.688503 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.688459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vv29"
Jul 27 16:18:08.732451 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.732408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:18:08.735588 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.735569    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 16:18:08.745892 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.745857    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 16:18:08.763494 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.763391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqrm\" (UniqueName: \"kubernetes.io/projected/1124f4b6-09d2-4b8b-9c50-46a34e10f6b8-kube-api-access-cbqrm\") pod \"network-check-target-m8nnm\" (UID: \"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8\") " pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:18:08.837329 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.837295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2vv29"]
Jul 27 16:18:08.840766 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:08.840734    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e88a58_768b_4d90_80f6_1cfc0a304f65.slice/crio-e8e0ae4745b5037d8b710fe55292e8f0c6e7bbb988c9741a882a9ae92c7cb112 WatchSource:0}: Error finding container e8e0ae4745b5037d8b710fe55292e8f0c6e7bbb988c9741a882a9ae92c7cb112: Status 404 returned error can't find the container with id e8e0ae4745b5037d8b710fe55292e8f0c6e7bbb988c9741a882a9ae92c7cb112
Jul 27 16:18:08.987521 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.987486    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fft22\""
Jul 27 16:18:08.995729 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:08.995698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:18:09.135416 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:09.135317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:18:09.135578 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:09.135430    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 16:18:09.135578 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:09.135496    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls podName:799b4fdb-c34f-434f-977d-323105f3709b nodeName:}" failed. No retries permitted until 2026-07-27 16:18:41.135481752 +0000 UTC m=+98.760662686 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls") pod "dns-default-pr7qc" (UID: "799b4fdb-c34f-434f-977d-323105f3709b") : secret "dns-default-metrics-tls" not found
Jul 27 16:18:09.162113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:09.162079    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m8nnm"]
Jul 27 16:18:09.165712 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:09.165682    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1124f4b6_09d2_4b8b_9c50_46a34e10f6b8.slice/crio-084b0f584b278cf56872a66b9dd0291c327f248c629ad5b78eaaf57dba787b71 WatchSource:0}: Error finding container 084b0f584b278cf56872a66b9dd0291c327f248c629ad5b78eaaf57dba787b71: Status 404 returned error can't find the container with id 084b0f584b278cf56872a66b9dd0291c327f248c629ad5b78eaaf57dba787b71
Jul 27 16:18:09.212397 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:09.212352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m8nnm" event={"ID":"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8","Type":"ContainerStarted","Data":"084b0f584b278cf56872a66b9dd0291c327f248c629ad5b78eaaf57dba787b71"}
Jul 27 16:18:09.213533 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:09.213499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2vv29" event={"ID":"09e88a58-768b-4d90-80f6-1cfc0a304f65","Type":"ContainerStarted","Data":"e8e0ae4745b5037d8b710fe55292e8f0c6e7bbb988c9741a882a9ae92c7cb112"}
Jul 27 16:18:12.222041 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:12.221920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m8nnm" event={"ID":"1124f4b6-09d2-4b8b-9c50-46a34e10f6b8","Type":"ContainerStarted","Data":"4a8e0249dec44dc297912b366961415720e36b4193c8a948f683c1f18b8823dc"}
Jul 27 16:18:12.222041 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:12.221983    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:18:12.246547 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:12.246494    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m8nnm" podStartSLOduration=66.443048538 podStartE2EDuration="1m9.246474785s" totalImagesPullingTime="2.803426247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:12.245380207 +0000 UTC m=+69.870561159" watchObservedRunningTime="2026-07-27 16:18:12.246474785 +0000 UTC m=+69.871655737"
Jul 27 16:18:16.235376 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:16.235334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2vv29" event={"ID":"09e88a58-768b-4d90-80f6-1cfc0a304f65","Type":"ContainerStarted","Data":"0cc79c9e344aecf0583115e920e5e73b2ba0b356929c6c554c18f09bec7d350b"}
Jul 27 16:18:16.251704 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:16.251626    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2vv29" podStartSLOduration=66.224601606 podStartE2EDuration="1m13.251607211s" totalImagesPullingTime="7.027005605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:16.251540528 +0000 UTC m=+73.876721483" watchObservedRunningTime="2026-07-27 16:18:16.251607211 +0000 UTC m=+73.876788165"
Jul 27 16:18:21.230924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.230867    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tw85l"]
Jul 27 16:18:21.233725 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.233698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.234155 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.234127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8669d845dc-k4ppl"]
Jul 27 16:18:21.236931 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.236913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.237535 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.237511    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:18:21.237643 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.237554    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 16:18:21.237643 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.237621    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 16:18:21.238695 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.238663    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 16:18:21.238797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.238759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9sfqv\""
Jul 27 16:18:21.240124 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.240106    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 27 16:18:21.240802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.240785    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5v8lj\""
Jul 27 16:18:21.241303 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.241273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 27 16:18:21.241303 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.241273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 16:18:21.242064 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.241943    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 27 16:18:21.242064 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.241958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 27 16:18:21.242583 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.242552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 16:18:21.242870 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.242846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 16:18:21.248207 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.248175    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tw85l"]
Jul 27 16:18:21.250436 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.250408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8669d845dc-k4ppl"]
Jul 27 16:18:21.323909 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.323865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-trusted-ca\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.324113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.323923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-config\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.324113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.323950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qprs\" (UniqueName: \"kubernetes.io/projected/cbe76343-1d2e-4430-adc8-c87ac6c49546-kube-api-access-7qprs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.324113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.324054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.324113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.324077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-stats-auth\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.324113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.324102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-default-certificate\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.324342 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.324122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.324342 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.324196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-serving-cert\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.324342 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.324243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwqwk\" (UniqueName: \"kubernetes.io/projected/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-kube-api-access-jwqwk\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.328434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.328402    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"]
Jul 27 16:18:21.331252 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.331226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"]
Jul 27 16:18:21.331369 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.331322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"
Jul 27 16:18:21.333765 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.333741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 16:18:21.334108 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.334084    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:18:21.334222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.334108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-crjwc\""
Jul 27 16:18:21.334320 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.334305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.337187 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.337168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:18:21.337319 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.337224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-k5dtx\""
Jul 27 16:18:21.337790 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.337721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 27 16:18:21.341871 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.341803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 27 16:18:21.342176 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.341993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 27 16:18:21.343501 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.343480    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"]
Jul 27 16:18:21.346374 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.346351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"]
Jul 27 16:18:21.425509 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.425509 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-stats-auth\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwxh\" (UniqueName: \"kubernetes.io/projected/0b311a84-36bf-4180-bdaa-b100e3f54e11-kube-api-access-lqwxh\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-default-certificate\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b311a84-36bf-4180-bdaa-b100e3f54e11-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-serving-cert\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwqwk\" (UniqueName: \"kubernetes.io/projected/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-kube-api-access-jwqwk\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-trusted-ca\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-config\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qprs\" (UniqueName: \"kubernetes.io/projected/cbe76343-1d2e-4430-adc8-c87ac6c49546-kube-api-access-7qprs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.425787 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dflrm\" (UniqueName: \"kubernetes.io/projected/23882235-2eb7-4b36-a10c-107bd62ac2e3-kube-api-access-dflrm\") pod \"volume-data-source-validator-549c6d54cf-qmfvb\" (UID: \"23882235-2eb7-4b36-a10c-107bd62ac2e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"
Jul 27 16:18:21.426285 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.425825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b311a84-36bf-4180-bdaa-b100e3f54e11-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.426285 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.425966    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:21.925950132 +0000 UTC m=+79.551131062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : configmap references non-existent config key: service-ca.crt
Jul 27 16:18:21.426285 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.426143    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 16:18:21.426465 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.426290    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:21.926266775 +0000 UTC m=+79.551447722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : secret "router-metrics-certs-default" not found
Jul 27 16:18:21.426849 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.426826    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84dd7688cc-qp465"]
Jul 27 16:18:21.427036 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.426995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-config\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.427566 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.427537    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-trusted-ca\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.428973 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.428943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-default-certificate\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.429276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.429258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-stats-auth\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.429319 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.429282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-serving-cert\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.430197 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.430181    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.432615 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.432592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 16:18:21.432757 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.432726    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 16:18:21.432818 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.432733    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9k8c5\""
Jul 27 16:18:21.432897 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.432881    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 16:18:21.437935 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.437902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qprs\" (UniqueName: \"kubernetes.io/projected/cbe76343-1d2e-4430-adc8-c87ac6c49546-kube-api-access-7qprs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.438499 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.438476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwqwk\" (UniqueName: \"kubernetes.io/projected/4eeec8f8-f1c6-4692-acab-1dfa7aca2143-kube-api-access-jwqwk\") pod \"console-operator-584dcb6b54-tw85l\" (UID: \"4eeec8f8-f1c6-4692-acab-1dfa7aca2143\") " pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.439693 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.439674    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 16:18:21.442458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.442432    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dd7688cc-qp465"]
Jul 27 16:18:21.527044 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.526916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-bound-sa-token\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527044 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.526956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-certificates\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-image-registry-private-configuration\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dflrm\" (UniqueName: \"kubernetes.io/projected/23882235-2eb7-4b36-a10c-107bd62ac2e3-kube-api-access-dflrm\") pod \"volume-data-source-validator-549c6d54cf-qmfvb\" (UID: \"23882235-2eb7-4b36-a10c-107bd62ac2e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"
Jul 27 16:18:21.527296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnwbj\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-kube-api-access-lnwbj\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b311a84-36bf-4180-bdaa-b100e3f54e11-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.527296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwxh\" (UniqueName: \"kubernetes.io/projected/0b311a84-36bf-4180-bdaa-b100e3f54e11-kube-api-access-lqwxh\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.527296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b311a84-36bf-4180-bdaa-b100e3f54e11-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.527569 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527569 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3355757-a5b4-40b9-8bfb-261753c3cc8d-ca-trust-extracted\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527569 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-trusted-ca\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527569 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-installation-pull-secrets\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.527830 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.527811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b311a84-36bf-4180-bdaa-b100e3f54e11-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.529843 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.529819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0b311a84-36bf-4180-bdaa-b100e3f54e11-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.535892 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.535865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwxh\" (UniqueName: \"kubernetes.io/projected/0b311a84-36bf-4180-bdaa-b100e3f54e11-kube-api-access-lqwxh\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh\" (UID: \"0b311a84-36bf-4180-bdaa-b100e3f54e11\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.536127 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.536105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dflrm\" (UniqueName: \"kubernetes.io/projected/23882235-2eb7-4b36-a10c-107bd62ac2e3-kube-api-access-dflrm\") pod \"volume-data-source-validator-549c6d54cf-qmfvb\" (UID: \"23882235-2eb7-4b36-a10c-107bd62ac2e3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"
Jul 27 16:18:21.545951 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.545917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:21.627826 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.627782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.627826 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.627823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3355757-a5b4-40b9-8bfb-261753c3cc8d-ca-trust-extracted\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.627842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-trusted-ca\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628118 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.627964    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:18:21.628118 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.627989    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd7688cc-qp465: secret "image-registry-tls" not found
Jul 27 16:18:21.628118 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.628099    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls podName:b3355757-a5b4-40b9-8bfb-261753c3cc8d nodeName:}" failed. No retries permitted until 2026-07-27 16:18:22.128074539 +0000 UTC m=+79.753255491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls") pod "image-registry-84dd7688cc-qp465" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d") : secret "image-registry-tls" not found
Jul 27 16:18:21.628311 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.628290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3355757-a5b4-40b9-8bfb-261753c3cc8d-ca-trust-extracted\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628363 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.628344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-installation-pull-secrets\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628411 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.628402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-bound-sa-token\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628461 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.628428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-certificates\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628901 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.628487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-image-registry-private-configuration\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.628901 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.628554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnwbj\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-kube-api-access-lnwbj\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.629364 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.629340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-certificates\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.629743 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.629615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-trusted-ca\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.631624 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.631590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-image-registry-private-configuration\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.631752 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.631651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-installation-pull-secrets\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.639572 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.639535    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-bound-sa-token\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.640107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.640090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnwbj\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-kube-api-access-lnwbj\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:21.646925 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.646890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"
Jul 27 16:18:21.650579 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.650556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"
Jul 27 16:18:21.680944 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.680904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-tw85l"]
Jul 27 16:18:21.684379 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:21.684343    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eeec8f8_f1c6_4692_acab_1dfa7aca2143.slice/crio-6e7aa1aad3b762fbd5bc20d95c291a9535f56bc9b3234ae4f858714e09b1c990 WatchSource:0}: Error finding container 6e7aa1aad3b762fbd5bc20d95c291a9535f56bc9b3234ae4f858714e09b1c990: Status 404 returned error can't find the container with id 6e7aa1aad3b762fbd5bc20d95c291a9535f56bc9b3234ae4f858714e09b1c990
Jul 27 16:18:21.795435 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.795318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb"]
Jul 27 16:18:21.799774 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:21.799732    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23882235_2eb7_4b36_a10c_107bd62ac2e3.slice/crio-c3f0ae62d3a12e59da2fbfcff582836e6756f597a6b144bfd1ca4df307626571 WatchSource:0}: Error finding container c3f0ae62d3a12e59da2fbfcff582836e6756f597a6b144bfd1ca4df307626571: Status 404 returned error can't find the container with id c3f0ae62d3a12e59da2fbfcff582836e6756f597a6b144bfd1ca4df307626571
Jul 27 16:18:21.811972 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.811894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh"]
Jul 27 16:18:21.814514 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:21.814483    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b311a84_36bf_4180_bdaa_b100e3f54e11.slice/crio-ae5695feacfcb2718c15b69253cdaaa89c85d9ea9f557fedb6ecd411e2044c71 WatchSource:0}: Error finding container ae5695feacfcb2718c15b69253cdaaa89c85d9ea9f557fedb6ecd411e2044c71: Status 404 returned error can't find the container with id ae5695feacfcb2718c15b69253cdaaa89c85d9ea9f557fedb6ecd411e2044c71
Jul 27 16:18:21.931336 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.931298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.931501 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:21.931362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:21.931501 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.931464    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 16:18:21.931597 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.931505    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:22.9314911 +0000 UTC m=+80.556672032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : configmap references non-existent config key: service-ca.crt
Jul 27 16:18:21.931597 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:21.931542    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:22.931526614 +0000 UTC m=+80.556707545 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : secret "router-metrics-certs-default" not found
Jul 27 16:18:22.133045 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:22.132921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:22.133191 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:22.133092    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:18:22.133191 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:22.133111    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd7688cc-qp465: secret "image-registry-tls" not found
Jul 27 16:18:22.133191 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:22.133168    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls podName:b3355757-a5b4-40b9-8bfb-261753c3cc8d nodeName:}" failed. No retries permitted until 2026-07-27 16:18:23.133152524 +0000 UTC m=+80.758333454 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls") pod "image-registry-84dd7688cc-qp465" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d") : secret "image-registry-tls" not found
Jul 27 16:18:22.249892 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:22.249853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh" event={"ID":"0b311a84-36bf-4180-bdaa-b100e3f54e11","Type":"ContainerStarted","Data":"ae5695feacfcb2718c15b69253cdaaa89c85d9ea9f557fedb6ecd411e2044c71"}
Jul 27 16:18:22.251064 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:22.251038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb" event={"ID":"23882235-2eb7-4b36-a10c-107bd62ac2e3","Type":"ContainerStarted","Data":"c3f0ae62d3a12e59da2fbfcff582836e6756f597a6b144bfd1ca4df307626571"}
Jul 27 16:18:22.252017 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:22.251983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" event={"ID":"4eeec8f8-f1c6-4692-acab-1dfa7aca2143","Type":"ContainerStarted","Data":"6e7aa1aad3b762fbd5bc20d95c291a9535f56bc9b3234ae4f858714e09b1c990"}
Jul 27 16:18:22.941443 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:22.940956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:22.941443 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:22.941103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:22.941443 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:22.941248    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 16:18:22.941443 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:22.941310    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:24.941290748 +0000 UTC m=+82.566471685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : secret "router-metrics-certs-default" not found
Jul 27 16:18:22.941964 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:22.941926    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:24.941903328 +0000 UTC m=+82.567084280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : configmap references non-existent config key: service-ca.crt
Jul 27 16:18:23.143702 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:23.143141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:23.143702 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:23.143259    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:18:23.143702 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:23.143277    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd7688cc-qp465: secret "image-registry-tls" not found
Jul 27 16:18:23.143702 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:23.143345    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls podName:b3355757-a5b4-40b9-8bfb-261753c3cc8d nodeName:}" failed. No retries permitted until 2026-07-27 16:18:25.1433225 +0000 UTC m=+82.768503431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls") pod "image-registry-84dd7688cc-qp465" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d") : secret "image-registry-tls" not found
Jul 27 16:18:24.262958 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:24.262912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb" event={"ID":"23882235-2eb7-4b36-a10c-107bd62ac2e3","Type":"ContainerStarted","Data":"78614d82eb5541701d05ca4bed3f59b40fa05d58ed430bc7ed4d42eba0c8f64d"}
Jul 27 16:18:24.280545 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:24.280485    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-qmfvb" podStartSLOduration=1.876913757 podStartE2EDuration="3.280468149s" totalImagesPullingTime="1.403554392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:24.279473737 +0000 UTC m=+81.904654695" watchObservedRunningTime="2026-07-27 16:18:24.280468149 +0000 UTC m=+81.905649144"
Jul 27 16:18:24.958830 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:24.958789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:24.959078 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:24.958868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:24.959078 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:24.958974    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 16:18:24.959078 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:24.958983    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.958969366 +0000 UTC m=+86.584150296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : configmap references non-existent config key: service-ca.crt
Jul 27 16:18:24.959263 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:24.959161    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.959144725 +0000 UTC m=+86.584325661 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : secret "router-metrics-certs-default" not found
Jul 27 16:18:25.161858 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.161808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:25.162078 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:25.161959    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:18:25.162078 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:25.161981    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd7688cc-qp465: secret "image-registry-tls" not found
Jul 27 16:18:25.162078 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:25.162070    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls podName:b3355757-a5b4-40b9-8bfb-261753c3cc8d nodeName:}" failed. No retries permitted until 2026-07-27 16:18:29.162053061 +0000 UTC m=+86.787233995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls") pod "image-registry-84dd7688cc-qp465" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d") : secret "image-registry-tls" not found
Jul 27 16:18:25.268469 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.268376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/0.log"
Jul 27 16:18:25.268469 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.268427    2579 generic.go:356] "Generic (PLEG): container finished" podID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143" containerID="cbfbbedf7ff7f0c8b2e0989a51f73ce786dd68db0d26c8b81ff9813017e3b07d" exitCode=255
Jul 27 16:18:25.268985 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.268517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" event={"ID":"4eeec8f8-f1c6-4692-acab-1dfa7aca2143","Type":"ContainerDied","Data":"cbfbbedf7ff7f0c8b2e0989a51f73ce786dd68db0d26c8b81ff9813017e3b07d"}
Jul 27 16:18:25.268985 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.268755    2579 scope.go:117] "RemoveContainer" containerID="cbfbbedf7ff7f0c8b2e0989a51f73ce786dd68db0d26c8b81ff9813017e3b07d"
Jul 27 16:18:25.270211 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.270179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh" event={"ID":"0b311a84-36bf-4180-bdaa-b100e3f54e11","Type":"ContainerStarted","Data":"efe134d9114300aa30b6a13207d77c8a6666e24b9ad94a8f994e7c2a60f6c417"}
Jul 27 16:18:25.307621 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:25.307571    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh" podStartSLOduration=1.515602731 podStartE2EDuration="4.307552628s" totalImagesPullingTime="2.791949897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:25.306701404 +0000 UTC m=+82.931882370" watchObservedRunningTime="2026-07-27 16:18:25.307552628 +0000 UTC m=+82.932733559"
Jul 27 16:18:26.278117 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.278090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/1.log"
Jul 27 16:18:26.278530 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.278514    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/0.log"
Jul 27 16:18:26.278579 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.278552    2579 generic.go:356] "Generic (PLEG): container finished" podID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143" containerID="43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d" exitCode=255
Jul 27 16:18:26.278667 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.278648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" event={"ID":"4eeec8f8-f1c6-4692-acab-1dfa7aca2143","Type":"ContainerDied","Data":"43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d"}
Jul 27 16:18:26.278736 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.278725    2579 scope.go:117] "RemoveContainer" containerID="cbfbbedf7ff7f0c8b2e0989a51f73ce786dd68db0d26c8b81ff9813017e3b07d"
Jul 27 16:18:26.278928 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.278899    2579 scope.go:117] "RemoveContainer" containerID="43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d"
Jul 27 16:18:26.279140 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:26.279119    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tw85l_openshift-console-operator(4eeec8f8-f1c6-4692-acab-1dfa7aca2143)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" podUID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143"
Jul 27 16:18:26.781564 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.781529    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5sqz4"]
Jul 27 16:18:26.785678 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.785652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.788939 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.788917    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 16:18:26.789475 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.789461    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 16:18:26.790033 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.790018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m7sx7\""
Jul 27 16:18:26.792626 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.792608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 16:18:26.794170 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.794153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 16:18:26.839151 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.839115    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5sqz4"]
Jul 27 16:18:26.875732 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.875701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f50b064e-fb6d-4492-8a3d-d873d8468524-crio-socket\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.875924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.875750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50b064e-fb6d-4492-8a3d-d873d8468524-data-volume\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.875924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.875788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.875924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.875809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j8zn\" (UniqueName: \"kubernetes.io/projected/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-api-access-9j8zn\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.875924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.875838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977131 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977131 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j8zn\" (UniqueName: \"kubernetes.io/projected/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-api-access-9j8zn\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:26.977236    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f50b064e-fb6d-4492-8a3d-d873d8468524-crio-socket\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:26.977321    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:27.47730377 +0000 UTC m=+85.102484701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f50b064e-fb6d-4492-8a3d-d873d8468524-crio-socket\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:26.977345    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:26.977390 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50b064e-fb6d-4492-8a3d-d873d8468524-data-volume\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.977633 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:26.977421    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:27.477401605 +0000 UTC m=+85.102582552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:26.977681 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.977643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f50b064e-fb6d-4492-8a3d-d873d8468524-data-volume\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:26.990419 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:26.990377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j8zn\" (UniqueName: \"kubernetes.io/projected/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-api-access-9j8zn\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:27.283952 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:27.283921    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/1.log"
Jul 27 16:18:27.284445 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:27.284398    2579 scope.go:117] "RemoveContainer" containerID="43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d"
Jul 27 16:18:27.284608 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:27.284581    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tw85l_openshift-console-operator(4eeec8f8-f1c6-4692-acab-1dfa7aca2143)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" podUID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143"
Jul 27 16:18:27.481356 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:27.481309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:27.481539 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:27.481438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:27.481539 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:27.481468    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:27.481539 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:27.481498    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:27.481640 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:27.481542    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.481521073 +0000 UTC m=+86.106702051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:27.481640 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:27.481559    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:28.481552793 +0000 UTC m=+86.106733724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:28.488570 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:28.488533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:28.488979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:28.488585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:28.488979 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.488680    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:28.488979 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.488700    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:28.488979 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.488747    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:30.488734683 +0000 UTC m=+88.113915614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:28.488979 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.488761    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:30.48875339 +0000 UTC m=+88.113934320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:28.993276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:28.993237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:28.993455 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:28.993311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:28.993455 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.993382    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 16:18:28.993455 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.993437    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:36.99342328 +0000 UTC m=+94.618604210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : configmap references non-existent config key: service-ca.crt
Jul 27 16:18:28.993566 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:28.993458    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs podName:cbe76343-1d2e-4430-adc8-c87ac6c49546 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:36.993449538 +0000 UTC m=+94.618630477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs") pod "router-default-8669d845dc-k4ppl" (UID: "cbe76343-1d2e-4430-adc8-c87ac6c49546") : secret "router-metrics-certs-default" not found
Jul 27 16:18:29.194580 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:29.194545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:29.194743 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:29.194716    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 16:18:29.194743 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:29.194731    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84dd7688cc-qp465: secret "image-registry-tls" not found
Jul 27 16:18:29.194832 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:29.194790    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls podName:b3355757-a5b4-40b9-8bfb-261753c3cc8d nodeName:}" failed. No retries permitted until 2026-07-27 16:18:37.194769926 +0000 UTC m=+94.819950861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls") pod "image-registry-84dd7688cc-qp465" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d") : secret "image-registry-tls" not found
Jul 27 16:18:29.514949 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:29.514921    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klbl8_a09ef6bc-7e7b-4803-bb17-325310a8cf7f/dns-node-resolver/0.log"
Jul 27 16:18:30.308167 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:30.308139    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-btrf8_dde7bb49-725e-45b3-b4bb-53fff8ad3516/node-ca/0.log"
Jul 27 16:18:30.508088 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:30.508026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:30.508278 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:30.508118    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:30.508278 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:30.508131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:30.508278 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:30.508202    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:34.508182465 +0000 UTC m=+92.133363420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:30.508278 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:30.508237    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:30.508417 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:30.508286    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:34.508275016 +0000 UTC m=+92.133455947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:31.510560 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:31.510535    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tx9vb_5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7/serve-healthcheck-canary/0.log"
Jul 27 16:18:31.546870 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:31.546842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:31.546975 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:31.546906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:31.547360 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:31.547338    2579 scope.go:117] "RemoveContainer" containerID="43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d"
Jul 27 16:18:31.547525 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:31.547504    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-tw85l_openshift-console-operator(4eeec8f8-f1c6-4692-acab-1dfa7aca2143)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" podUID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143"
Jul 27 16:18:32.108752 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:32.108717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh_0b311a84-36bf-4180-bdaa-b100e3f54e11/kube-storage-version-migrator-operator/0.log"
Jul 27 16:18:34.540924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:34.540869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:34.541475 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:34.540949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:34.541475 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:34.541020    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 16:18:34.541475 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:34.541097    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:42.541081646 +0000 UTC m=+100.166262577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : configmap "kube-rbac-proxy" not found
Jul 27 16:18:34.541475 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:34.541121    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:34.541475 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:34.541191    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls podName:f50b064e-fb6d-4492-8a3d-d873d8468524 nodeName:}" failed. No retries permitted until 2026-07-27 16:18:42.541175868 +0000 UTC m=+100.166356817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5sqz4" (UID: "f50b064e-fb6d-4492-8a3d-d873d8468524") : secret "insights-runtime-extractor-tls" not found
Jul 27 16:18:37.061946 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.061899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:37.062353 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.062037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:37.062655 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.062635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbe76343-1d2e-4430-adc8-c87ac6c49546-service-ca-bundle\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:37.064579 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.064550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe76343-1d2e-4430-adc8-c87ac6c49546-metrics-certs\") pod \"router-default-8669d845dc-k4ppl\" (UID: \"cbe76343-1d2e-4430-adc8-c87ac6c49546\") " pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:37.153232 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.153180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:37.264976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.264940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:37.267578 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.267555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"image-registry-84dd7688cc-qp465\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") " pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:37.320674 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.320522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8669d845dc-k4ppl"]
Jul 27 16:18:37.324058 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:37.324029    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe76343_1d2e_4430_adc8_c87ac6c49546.slice/crio-284efa18d7398ec9977c4223cf46f964c120006e7e35c3cbcb1b3b54cf1e4a6d WatchSource:0}: Error finding container 284efa18d7398ec9977c4223cf46f964c120006e7e35c3cbcb1b3b54cf1e4a6d: Status 404 returned error can't find the container with id 284efa18d7398ec9977c4223cf46f964c120006e7e35c3cbcb1b3b54cf1e4a6d
Jul 27 16:18:37.354166 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.354123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:37.529487 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:37.529450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84dd7688cc-qp465"]
Jul 27 16:18:37.533605 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:37.533575    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3355757_a5b4_40b9_8bfb_261753c3cc8d.slice/crio-07f7e5bb48644c4d0e88cee857f61b506f04405f8284a2d214bf26888451962a WatchSource:0}: Error finding container 07f7e5bb48644c4d0e88cee857f61b506f04405f8284a2d214bf26888451962a: Status 404 returned error can't find the container with id 07f7e5bb48644c4d0e88cee857f61b506f04405f8284a2d214bf26888451962a
Jul 27 16:18:38.316949 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.316901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd7688cc-qp465" event={"ID":"b3355757-a5b4-40b9-8bfb-261753c3cc8d","Type":"ContainerStarted","Data":"6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda"}
Jul 27 16:18:38.317446 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.316977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:18:38.317446 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.317023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd7688cc-qp465" event={"ID":"b3355757-a5b4-40b9-8bfb-261753c3cc8d","Type":"ContainerStarted","Data":"07f7e5bb48644c4d0e88cee857f61b506f04405f8284a2d214bf26888451962a"}
Jul 27 16:18:38.318482 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.318457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8669d845dc-k4ppl" event={"ID":"cbe76343-1d2e-4430-adc8-c87ac6c49546","Type":"ContainerStarted","Data":"d37b1266e2aadeb7ee004ceedfc85de4de51a130619a8243622ef363ca679484"}
Jul 27 16:18:38.318587 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.318499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8669d845dc-k4ppl" event={"ID":"cbe76343-1d2e-4430-adc8-c87ac6c49546","Type":"ContainerStarted","Data":"284efa18d7398ec9977c4223cf46f964c120006e7e35c3cbcb1b3b54cf1e4a6d"}
Jul 27 16:18:38.350679 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.350618    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84dd7688cc-qp465" podStartSLOduration=17.350601595 podStartE2EDuration="17.350601595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:38.348903194 +0000 UTC m=+95.974084147" watchObservedRunningTime="2026-07-27 16:18:38.350601595 +0000 UTC m=+95.975782549"
Jul 27 16:18:38.385243 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:38.385181    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8669d845dc-k4ppl" podStartSLOduration=17.385166849 podStartE2EDuration="17.385166849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:38.384476289 +0000 UTC m=+96.009657254" watchObservedRunningTime="2026-07-27 16:18:38.385166849 +0000 UTC m=+96.010347802"
Jul 27 16:18:39.154262 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:39.154222    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:39.157089 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:39.157061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:39.325545 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:39.325511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:39.326780 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:39.326755    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8669d845dc-k4ppl"
Jul 27 16:18:41.200594 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:41.200539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:18:41.203054 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:41.203031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/799b4fdb-c34f-434f-977d-323105f3709b-metrics-tls\") pod \"dns-default-pr7qc\" (UID: \"799b4fdb-c34f-434f-977d-323105f3709b\") " pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:18:41.501084 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:41.501055    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2mnq\""
Jul 27 16:18:41.508300 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:41.508264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:18:41.661432 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:41.661401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pr7qc"]
Jul 27 16:18:41.664543 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:41.664513    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799b4fdb_c34f_434f_977d_323105f3709b.slice/crio-d32203e96dbaef02413444620ab86633dd47ad52461d80e485a7ead0f8ccd74b WatchSource:0}: Error finding container d32203e96dbaef02413444620ab86633dd47ad52461d80e485a7ead0f8ccd74b: Status 404 returned error can't find the container with id d32203e96dbaef02413444620ab86633dd47ad52461d80e485a7ead0f8ccd74b
Jul 27 16:18:41.962247 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:41.962222    2579 scope.go:117] "RemoveContainer" containerID="43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d"
Jul 27 16:18:42.337723 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.337628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pr7qc" event={"ID":"799b4fdb-c34f-434f-977d-323105f3709b","Type":"ContainerStarted","Data":"d32203e96dbaef02413444620ab86633dd47ad52461d80e485a7ead0f8ccd74b"}
Jul 27 16:18:42.339558 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.339536    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:18:42.340053 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.340033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/1.log"
Jul 27 16:18:42.340150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.340080    2579 generic.go:356] "Generic (PLEG): container finished" podID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143" containerID="3b9cab50e535b662ca6dc1e4b24a1b5c62a09eac96bb2bc544962c97ca67d528" exitCode=255
Jul 27 16:18:42.340150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.340138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" event={"ID":"4eeec8f8-f1c6-4692-acab-1dfa7aca2143","Type":"ContainerDied","Data":"3b9cab50e535b662ca6dc1e4b24a1b5c62a09eac96bb2bc544962c97ca67d528"}
Jul 27 16:18:42.340235 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.340199    2579 scope.go:117] "RemoveContainer" containerID="43134972d148de1df647ecaa13e12b77ab82fe7d04d3ec5cf6cb31b3c9b47c6d"
Jul 27 16:18:42.341055 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.340542    2579 scope.go:117] "RemoveContainer" containerID="3b9cab50e535b662ca6dc1e4b24a1b5c62a09eac96bb2bc544962c97ca67d528"
Jul 27 16:18:42.341055 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:42.340781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-tw85l_openshift-console-operator(4eeec8f8-f1c6-4692-acab-1dfa7aca2143)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" podUID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143"
Jul 27 16:18:42.614639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.614540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:42.614639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.614618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:42.615209 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.615178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f50b064e-fb6d-4492-8a3d-d873d8468524-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:42.617559 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.617530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f50b064e-fb6d-4492-8a3d-d873d8468524-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5sqz4\" (UID: \"f50b064e-fb6d-4492-8a3d-d873d8468524\") " pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:42.694389 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.694346    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5sqz4"
Jul 27 16:18:42.881898 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:42.881793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5sqz4"]
Jul 27 16:18:42.885769 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:42.885744    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50b064e_fb6d_4492_8a3d_d873d8468524.slice/crio-4c6280c9d3f2c3bbaea67380258a2be68cade4bb7495e937161f17bfc2cafbca WatchSource:0}: Error finding container 4c6280c9d3f2c3bbaea67380258a2be68cade4bb7495e937161f17bfc2cafbca: Status 404 returned error can't find the container with id 4c6280c9d3f2c3bbaea67380258a2be68cade4bb7495e937161f17bfc2cafbca
Jul 27 16:18:43.228410 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:43.228375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m8nnm"
Jul 27 16:18:43.345744 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:43.345712    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:18:43.347207 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:43.347177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sqz4" event={"ID":"f50b064e-fb6d-4492-8a3d-d873d8468524","Type":"ContainerStarted","Data":"4c6280c9d3f2c3bbaea67380258a2be68cade4bb7495e937161f17bfc2cafbca"}
Jul 27 16:18:44.352499 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:44.352460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pr7qc" event={"ID":"799b4fdb-c34f-434f-977d-323105f3709b","Type":"ContainerStarted","Data":"1d78152f8a76342f73f7248b14ae77d1d4004402cb7f4bb52e2be8862839c9f8"}
Jul 27 16:18:44.352914 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:44.352523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pr7qc" event={"ID":"799b4fdb-c34f-434f-977d-323105f3709b","Type":"ContainerStarted","Data":"1c67d0f6f4726cbadca3e835687d8f3bbb5781c724b0c5d5d338d24a9308f958"}
Jul 27 16:18:44.352914 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:44.352570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:18:44.380624 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:44.380562    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pr7qc" podStartSLOduration=65.446241286 podStartE2EDuration="1m7.380542227s" totalImagesPullingTime="1.934300941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:44.37980825 +0000 UTC m=+102.004989216" watchObservedRunningTime="2026-07-27 16:18:44.380542227 +0000 UTC m=+102.005723179"
Jul 27 16:18:50.018690 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.018642    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84dd7688cc-qp465"]
Jul 27 16:18:50.092985 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.092949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"]
Jul 27 16:18:50.172107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.172072    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"]
Jul 27 16:18:50.172107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.172093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.280963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-bound-sa-token\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-registry-tls\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvssr\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-kube-api-access-zvssr\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281114    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a107051-009c-43cb-84f3-26a86f62a3e9-registry-certificates\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a107051-009c-43cb-84f3-26a86f62a3e9-installation-pull-secrets\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a107051-009c-43cb-84f3-26a86f62a3e9-image-registry-private-configuration\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a107051-009c-43cb-84f3-26a86f62a3e9-trusted-ca\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.281420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.281312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a107051-009c-43cb-84f3-26a86f62a3e9-ca-trust-extracted\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.382538 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a107051-009c-43cb-84f3-26a86f62a3e9-ca-trust-extracted\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.382747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-bound-sa-token\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.382747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-registry-tls\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.382747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvssr\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-kube-api-access-zvssr\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.382747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a107051-009c-43cb-84f3-26a86f62a3e9-registry-certificates\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.382747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a107051-009c-43cb-84f3-26a86f62a3e9-installation-pull-secrets\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.383051 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a107051-009c-43cb-84f3-26a86f62a3e9-image-registry-private-configuration\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.383051 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382783    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a107051-009c-43cb-84f3-26a86f62a3e9-trusted-ca\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.383051 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.382959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9a107051-009c-43cb-84f3-26a86f62a3e9-ca-trust-extracted\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.383802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.383753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9a107051-009c-43cb-84f3-26a86f62a3e9-trusted-ca\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.383802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.383776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9a107051-009c-43cb-84f3-26a86f62a3e9-registry-certificates\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.385763 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.385736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-registry-tls\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.386230 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.386202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9a107051-009c-43cb-84f3-26a86f62a3e9-installation-pull-secrets\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.386620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.386598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9a107051-009c-43cb-84f3-26a86f62a3e9-image-registry-private-configuration\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.407652 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.407616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvssr\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-kube-api-access-zvssr\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.414213 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.414184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9a107051-009c-43cb-84f3-26a86f62a3e9-bound-sa-token\") pod \"image-registry-7d9fc4f64d-fxjtl\" (UID: \"9a107051-009c-43cb-84f3-26a86f62a3e9\") " pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:50.484127 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:50.484088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:51.515052 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:51.514991    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"]
Jul 27 16:18:51.518149 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:18:51.518116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a107051_009c_43cb_84f3_26a86f62a3e9.slice/crio-d6c39f6944ec569b5a7f24e65e3ff5b8aa3dbbe3640a095327ad1d235351fed6 WatchSource:0}: Error finding container d6c39f6944ec569b5a7f24e65e3ff5b8aa3dbbe3640a095327ad1d235351fed6: Status 404 returned error can't find the container with id d6c39f6944ec569b5a7f24e65e3ff5b8aa3dbbe3640a095327ad1d235351fed6
Jul 27 16:18:51.546142 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:51.546103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:51.546321 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:51.546171    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:18:51.546530 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:51.546516    2579 scope.go:117] "RemoveContainer" containerID="3b9cab50e535b662ca6dc1e4b24a1b5c62a09eac96bb2bc544962c97ca67d528"
Jul 27 16:18:51.546689 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:18:51.546670    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-tw85l_openshift-console-operator(4eeec8f8-f1c6-4692-acab-1dfa7aca2143)\"" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" podUID="4eeec8f8-f1c6-4692-acab-1dfa7aca2143"
Jul 27 16:18:52.382085 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:52.382042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sqz4" event={"ID":"f50b064e-fb6d-4492-8a3d-d873d8468524","Type":"ContainerStarted","Data":"25d53f96f4a2ad2574ca6e67291fbea552ae0c6ae7c0ce59faac5c86819588bc"}
Jul 27 16:18:52.383769 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:52.383735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl" event={"ID":"9a107051-009c-43cb-84f3-26a86f62a3e9","Type":"ContainerStarted","Data":"4c9f45886111b8c15793d740919e5205e55d997c0010f7421fc00e6ecb6abd39"}
Jul 27 16:18:52.383935 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:52.383786    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl" event={"ID":"9a107051-009c-43cb-84f3-26a86f62a3e9","Type":"ContainerStarted","Data":"d6c39f6944ec569b5a7f24e65e3ff5b8aa3dbbe3640a095327ad1d235351fed6"}
Jul 27 16:18:52.383935 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:52.383874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:18:52.421243 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:52.421175    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl" podStartSLOduration=2.421149474 podStartE2EDuration="2.421149474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:52.420225739 +0000 UTC m=+110.045406693" watchObservedRunningTime="2026-07-27 16:18:52.421149474 +0000 UTC m=+110.046330428"
Jul 27 16:18:54.357963 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:54.357920    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pr7qc"
Jul 27 16:18:55.396453 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:55.396417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sqz4" event={"ID":"f50b064e-fb6d-4492-8a3d-d873d8468524","Type":"ContainerStarted","Data":"25dc740e2489d73c856eb7a783fa39ef943da3d65e3a54d3bc37648179914273"}
Jul 27 16:18:59.413410 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:59.413367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5sqz4" event={"ID":"f50b064e-fb6d-4492-8a3d-d873d8468524","Type":"ContainerStarted","Data":"1fd83aa110d4fef0ab82d630c204ab909b07ec0803be4ac4e7ddadd70673b2c3"}
Jul 27 16:18:59.434563 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:18:59.434509    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5sqz4" podStartSLOduration=17.594829472 podStartE2EDuration="33.434493479s" totalImagesPullingTime="15.839664007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:18:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:18:59.433039999 +0000 UTC m=+117.058220956" watchObservedRunningTime="2026-07-27 16:18:59.434493479 +0000 UTC m=+117.059674431"
Jul 27 16:19:00.024465 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:00.024438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:19:03.712463 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.712425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"]
Jul 27 16:19:03.716836 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.716815    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.719926 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.719899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-phl4k\""
Jul 27 16:19:03.721021 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.720975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 16:19:03.721021 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.721023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 16:19:03.721225 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.721028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 16:19:03.721225 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.720991    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 16:19:03.721225 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.721041    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 16:19:03.729797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.729772    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"]
Jul 27 16:19:03.733050 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.733021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"]
Jul 27 16:19:03.733205 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.733133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.735789 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.735767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tch8p\""
Jul 27 16:19:03.735939 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.735891    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 16:19:03.736025 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.735944    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 16:19:03.736076 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.735944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 16:19:03.746622 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.746592    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8hslt"]
Jul 27 16:19:03.750264 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.750231    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"]
Jul 27 16:19:03.750425 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.750358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.753089 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.753063    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 16:19:03.753809 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.753788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wzslv\""
Jul 27 16:19:03.754109 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.754091    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 16:19:03.754278 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.754258    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 16:19:03.793363 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-root\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793576 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4749b905-2dfa-4408-b1a9-d66436bf5b25-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.793576 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65b258f3-467c-4a00-9f74-7eda7c895684-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.793576 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.793576 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-wtmp\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793576 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swq6p\" (UniqueName: \"kubernetes.io/projected/4234a8fe-a8f2-4e81-8569-960422de412e-kube-api-access-swq6p\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793576 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58sg\" (UniqueName: \"kubernetes.io/projected/65b258f3-467c-4a00-9f74-7eda7c895684-kube-api-access-d58sg\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-tls\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4234a8fe-a8f2-4e81-8569-960422de412e-metrics-client-ca\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4749b905-2dfa-4408-b1a9-d66436bf5b25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4749b905-2dfa-4408-b1a9-d66436bf5b25-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-textfile\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65b258f3-467c-4a00-9f74-7eda7c895684-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.793896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-sys\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.794326 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lhz\" (UniqueName: \"kubernetes.io/projected/4749b905-2dfa-4408-b1a9-d66436bf5b25-kube-api-access-s6lhz\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.794326 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.793978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-root\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4749b905-2dfa-4408-b1a9-d66436bf5b25-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65b258f3-467c-4a00-9f74-7eda7c895684-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-root\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-wtmp\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swq6p\" (UniqueName: \"kubernetes.io/projected/4234a8fe-a8f2-4e81-8569-960422de412e-kube-api-access-swq6p\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58sg\" (UniqueName: \"kubernetes.io/projected/65b258f3-467c-4a00-9f74-7eda7c895684-kube-api-access-d58sg\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-tls\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-wtmp\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4234a8fe-a8f2-4e81-8569-960422de412e-metrics-client-ca\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4749b905-2dfa-4408-b1a9-d66436bf5b25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4749b905-2dfa-4408-b1a9-d66436bf5b25-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/65b258f3-467c-4a00-9f74-7eda7c895684-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-textfile\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65b258f3-467c-4a00-9f74-7eda7c895684-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-sys\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.895872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.895813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lhz\" (UniqueName: \"kubernetes.io/projected/4749b905-2dfa-4408-b1a9-d66436bf5b25-kube-api-access-s6lhz\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.896447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.896089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4749b905-2dfa-4408-b1a9-d66436bf5b25-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.896447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.896106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.896447 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:19:03.896214    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 16:19:03.896447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.896220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4234a8fe-a8f2-4e81-8569-960422de412e-metrics-client-ca\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.896447 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:19:03.896283    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-tls podName:4234a8fe-a8f2-4e81-8569-960422de412e nodeName:}" failed. No retries permitted until 2026-07-27 16:19:04.396264945 +0000 UTC m=+122.021445880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-tls") pod "node-exporter-8hslt" (UID: "4234a8fe-a8f2-4e81-8569-960422de412e") : secret "node-exporter-tls" not found
Jul 27 16:19:03.896447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.896357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-textfile\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.896916 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.896595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4234a8fe-a8f2-4e81-8569-960422de412e-sys\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.897130 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.897102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.897630 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.897603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65b258f3-467c-4a00-9f74-7eda7c895684-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.898226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.898204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:03.898407 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.898387    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4749b905-2dfa-4408-b1a9-d66436bf5b25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.898702 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.898681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4749b905-2dfa-4408-b1a9-d66436bf5b25-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.899084 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.899066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.899282 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.899262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/65b258f3-467c-4a00-9f74-7eda7c895684-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.909914 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.909823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58sg\" (UniqueName: \"kubernetes.io/projected/65b258f3-467c-4a00-9f74-7eda7c895684-kube-api-access-d58sg\") pod \"kube-state-metrics-56d9f944b4-ck5sc\" (UID: \"65b258f3-467c-4a00-9f74-7eda7c895684\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:03.910277 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.910255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lhz\" (UniqueName: \"kubernetes.io/projected/4749b905-2dfa-4408-b1a9-d66436bf5b25-kube-api-access-s6lhz\") pod \"openshift-state-metrics-b8f49f5c6-98wv5\" (UID: \"4749b905-2dfa-4408-b1a9-d66436bf5b25\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:03.910362 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:03.910277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swq6p\" (UniqueName: \"kubernetes.io/projected/4234a8fe-a8f2-4e81-8569-960422de412e-kube-api-access-swq6p\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:04.026404 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.026319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"
Jul 27 16:19:04.045762 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.045728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"
Jul 27 16:19:04.181441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.181399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5"]
Jul 27 16:19:04.185552 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:19:04.185524    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4749b905_2dfa_4408_b1a9_d66436bf5b25.slice/crio-00a9375e7c8991449b8d35d2d632db205f5e5803551f8f270b3d2c65688cfbdc WatchSource:0}: Error finding container 00a9375e7c8991449b8d35d2d632db205f5e5803551f8f270b3d2c65688cfbdc: Status 404 returned error can't find the container with id 00a9375e7c8991449b8d35d2d632db205f5e5803551f8f270b3d2c65688cfbdc
Jul 27 16:19:04.197578 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.197547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc"]
Jul 27 16:19:04.200534 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:19:04.200471    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b258f3_467c_4a00_9f74_7eda7c895684.slice/crio-9e2a5f22bf833aeaae3b306dd39a3633876bea64d58e5bcf27adbd842bfbea4e WatchSource:0}: Error finding container 9e2a5f22bf833aeaae3b306dd39a3633876bea64d58e5bcf27adbd842bfbea4e: Status 404 returned error can't find the container with id 9e2a5f22bf833aeaae3b306dd39a3633876bea64d58e5bcf27adbd842bfbea4e
Jul 27 16:19:04.400125 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.400087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-tls\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:04.402696 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.402661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4234a8fe-a8f2-4e81-8569-960422de412e-node-exporter-tls\") pod \"node-exporter-8hslt\" (UID: \"4234a8fe-a8f2-4e81-8569-960422de412e\") " pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:04.430650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.430611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5" event={"ID":"4749b905-2dfa-4408-b1a9-d66436bf5b25","Type":"ContainerStarted","Data":"92f49869029c771cdb723da3ea4472079cce16df0eb95bf2077fd01491b5aaa1"}
Jul 27 16:19:04.430806 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.430683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5" event={"ID":"4749b905-2dfa-4408-b1a9-d66436bf5b25","Type":"ContainerStarted","Data":"304a795cf6bce4cc6065b13df2cd4220c3b1081a93a94c02f38f5c6880caa4d4"}
Jul 27 16:19:04.430806 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.430711    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5" event={"ID":"4749b905-2dfa-4408-b1a9-d66436bf5b25","Type":"ContainerStarted","Data":"00a9375e7c8991449b8d35d2d632db205f5e5803551f8f270b3d2c65688cfbdc"}
Jul 27 16:19:04.431732 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.431706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc" event={"ID":"65b258f3-467c-4a00-9f74-7eda7c895684","Type":"ContainerStarted","Data":"9e2a5f22bf833aeaae3b306dd39a3633876bea64d58e5bcf27adbd842bfbea4e"}
Jul 27 16:19:04.660384 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:04.660280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hslt"
Jul 27 16:19:04.671134 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:19:04.671101    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4234a8fe_a8f2_4e81_8569_960422de412e.slice/crio-fcc49f07936594894691e6f89cec4a9a4334c61371be3ce2acc268b681c94431 WatchSource:0}: Error finding container fcc49f07936594894691e6f89cec4a9a4334c61371be3ce2acc268b681c94431: Status 404 returned error can't find the container with id fcc49f07936594894691e6f89cec4a9a4334c61371be3ce2acc268b681c94431
Jul 27 16:19:05.437499 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:05.437463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hslt" event={"ID":"4234a8fe-a8f2-4e81-8569-960422de412e","Type":"ContainerStarted","Data":"fcc49f07936594894691e6f89cec4a9a4334c61371be3ce2acc268b681c94431"}
Jul 27 16:19:06.442800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:06.442733    2579 generic.go:356] "Generic (PLEG): container finished" podID="4234a8fe-a8f2-4e81-8569-960422de412e" containerID="163186a4406400fda575764b269201fac37beb211a60fe20cbe5d5eb44a7296a" exitCode=0
Jul 27 16:19:06.443345 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:06.442828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hslt" event={"ID":"4234a8fe-a8f2-4e81-8569-960422de412e","Type":"ContainerDied","Data":"163186a4406400fda575764b269201fac37beb211a60fe20cbe5d5eb44a7296a"}
Jul 27 16:19:06.445320 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:06.445290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5" event={"ID":"4749b905-2dfa-4408-b1a9-d66436bf5b25","Type":"ContainerStarted","Data":"66ff02f8012ba5acb1617f6c83a56ac67012c23ea6cb8bd6aa15600dd407403a"}
Jul 27 16:19:06.495451 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:06.495348    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-98wv5" podStartSLOduration=1.611919959 podStartE2EDuration="3.495327508s" totalImagesPullingTime="1.883407549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:06.494228415 +0000 UTC m=+124.119409381" watchObservedRunningTime="2026-07-27 16:19:06.495327508 +0000 UTC m=+124.120508463"
Jul 27 16:19:06.961633 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:06.961600    2579 scope.go:117] "RemoveContainer" containerID="3b9cab50e535b662ca6dc1e4b24a1b5c62a09eac96bb2bc544962c97ca67d528"
Jul 27 16:19:07.453557 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.453513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hslt" event={"ID":"4234a8fe-a8f2-4e81-8569-960422de412e","Type":"ContainerStarted","Data":"001345a17ca44c5e7f48d92884585da2c3f159a19c8b581b27eb40ea3c257232"}
Jul 27 16:19:07.454082 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.453593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hslt" event={"ID":"4234a8fe-a8f2-4e81-8569-960422de412e","Type":"ContainerStarted","Data":"f73069f644e48a12d993cc3d42b8f61a59450eba55de8a4f3cb7627ba48d2d0d"}
Jul 27 16:19:07.456131 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.456093    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:19:07.456270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.456237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" event={"ID":"4eeec8f8-f1c6-4692-acab-1dfa7aca2143","Type":"ContainerStarted","Data":"3c858e44db2ade6e4ab424e0e5a925b5bc536eb09e5d3de5fb6ab13b3b47e49b"}
Jul 27 16:19:07.456795 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.456764    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:19:07.462993 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.462958    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l"
Jul 27 16:19:07.485684 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.485627    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8hslt" podStartSLOduration=3.5344417999999997 podStartE2EDuration="4.485610008s" totalImagesPullingTime="875.143114ms" totalInitContainerRuntime="76.025094ms" isStatefulPod=true podCreationTimestamp="2026-07-27 16:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:07.484217855 +0000 UTC m=+125.109398808" watchObservedRunningTime="2026-07-27 16:19:07.485610008 +0000 UTC m=+125.110790939"
Jul 27 16:19:07.506727 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:07.506668    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-tw85l" podStartSLOduration=43.582428499 podStartE2EDuration="46.506649862s" totalImagesPullingTime="2.924221363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:07.505854648 +0000 UTC m=+125.131035600" watchObservedRunningTime="2026-07-27 16:19:07.506649862 +0000 UTC m=+125.131830816"
Jul 27 16:19:08.916435 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.916401    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55bdd94856-rvs42"]
Jul 27 16:19:08.921020 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.920979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.923720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.923665    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 16:19:08.923881 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.923762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 16:19:08.923992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.923978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 16:19:08.924314 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.924294    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 16:19:08.924386 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.924294    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-l9jt4\""
Jul 27 16:19:08.925079 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.925019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 16:19:08.929059 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.929039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 16:19:08.932668 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.932643    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55bdd94856-rvs42"]
Jul 27 16:19:08.938903 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.938871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-serving-certs-ca-bundle\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939066 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.938911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-metrics-client-ca\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939066 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.938931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5fvn\" (UniqueName: \"kubernetes.io/projected/f71e5436-f3aa-4a52-ac4d-c91319a29420-kube-api-access-k5fvn\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939066 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.938977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-secret-telemeter-client\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939203 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.939090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939203 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.939154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-telemeter-client-tls\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939203 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.939179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-federate-client-tls\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:08.939334 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:08.939203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.040650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-telemeter-client-tls\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.040650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-federate-client-tls\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.040650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.040650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-serving-certs-ca-bundle\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.040650 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-metrics-client-ca\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.041090 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5fvn\" (UniqueName: \"kubernetes.io/projected/f71e5436-f3aa-4a52-ac4d-c91319a29420-kube-api-access-k5fvn\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.041090 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-secret-telemeter-client\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.041090 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.040759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.041934 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.041903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.042701 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.042671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-serving-certs-ca-bundle\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.043124 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.043095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f71e5436-f3aa-4a52-ac4d-c91319a29420-metrics-client-ca\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.045503 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.045449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-telemeter-client-tls\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.046871 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.046374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.046871 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.046834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-federate-client-tls\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.047241 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.047072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f71e5436-f3aa-4a52-ac4d-c91319a29420-secret-telemeter-client\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.058275 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.058239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5fvn\" (UniqueName: \"kubernetes.io/projected/f71e5436-f3aa-4a52-ac4d-c91319a29420-kube-api-access-k5fvn\") pod \"telemeter-client-55bdd94856-rvs42\" (UID: \"f71e5436-f3aa-4a52-ac4d-c91319a29420\") " pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.231100 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.231055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42"
Jul 27 16:19:09.386165 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.386127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55bdd94856-rvs42"]
Jul 27 16:19:09.389088 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:19:09.389050    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71e5436_f3aa_4a52_ac4d_c91319a29420.slice/crio-40c1a21ffd2a6b32f1e44288036524fe8b76343c8d8d883f7386071e0373d939 WatchSource:0}: Error finding container 40c1a21ffd2a6b32f1e44288036524fe8b76343c8d8d883f7386071e0373d939: Status 404 returned error can't find the container with id 40c1a21ffd2a6b32f1e44288036524fe8b76343c8d8d883f7386071e0373d939
Jul 27 16:19:09.466705 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.466658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc" event={"ID":"65b258f3-467c-4a00-9f74-7eda7c895684","Type":"ContainerStarted","Data":"166380e8d1c805aee330c0537729243efe786b1e801bd8fa93e04cc585e1820e"}
Jul 27 16:19:09.466911 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.466741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc" event={"ID":"65b258f3-467c-4a00-9f74-7eda7c895684","Type":"ContainerStarted","Data":"14729bc77373097a422a21882eee49579d0a4c8ec362683b58747b1731003d7a"}
Jul 27 16:19:09.466911 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.466772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc" event={"ID":"65b258f3-467c-4a00-9f74-7eda7c895684","Type":"ContainerStarted","Data":"64927158e503e52ee3f304df5c35811b64856de97a1afcf53bdf61ac8c81821e"}
Jul 27 16:19:09.467828 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.467800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42" event={"ID":"f71e5436-f3aa-4a52-ac4d-c91319a29420","Type":"ContainerStarted","Data":"40c1a21ffd2a6b32f1e44288036524fe8b76343c8d8d883f7386071e0373d939"}
Jul 27 16:19:09.487349 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:09.487205    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-ck5sc" podStartSLOduration=1.813758988 podStartE2EDuration="6.487183273s" totalImagesPullingTime="4.673424285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:09.486872709 +0000 UTC m=+127.112053662" watchObservedRunningTime="2026-07-27 16:19:09.487183273 +0000 UTC m=+127.112364226"
Jul 27 16:19:10.016110 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.016070    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:19:10.021378 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.021350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.024139 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.024110    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 16:19:10.025247 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 16:19:10.025247 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025231    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 16:19:10.025450 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025251    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 16:19:10.025450 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 16:19:10.025450 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 16:19:10.025450 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025228    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 16:19:10.025450 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025368    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 16:19:10.025775 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 16:19:10.025775 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 16:19:10.025947 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fd55od0fqc1ge\""
Jul 27 16:19:10.025947 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 16:19:10.025947 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025812    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 16:19:10.025947 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025856    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 16:19:10.026166 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.025954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 16:19:10.026760 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.026742    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zrn5q\""
Jul 27 16:19:10.029849 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.029815    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 16:19:10.038407 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.038369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:19:10.052282 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052565 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052606 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052657 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052825 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052825 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cxqm\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-kube-api-access-6cxqm\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052825 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052970 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052970 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052970 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-config\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052970 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.052970 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.052937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.053198 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.053049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.053198 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.053088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.053198 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.053123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.153971 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.153922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.153971 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.153975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-config\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.153993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cxqm\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-kube-api-access-6cxqm\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154516    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.154751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.154722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.155465 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.155430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.155567 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.155439    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.157857 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.155945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.157857 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.157155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.157857 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.157528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.158343 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.158314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.158603 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.158573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.159111 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.159085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.159433 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.159405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.159520 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.159478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.159607 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.159580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.160191 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.160166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-config\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.160755 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.160722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.160755 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.160743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-web-config\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.161217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.161198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.161309 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.161223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.161427 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.161410    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-config-out\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.167106 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.166983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cxqm\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-kube-api-access-6cxqm\") pod \"prometheus-k8s-0\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.333877 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.333775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:10.521653 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:10.521617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:19:10.525339 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:19:10.525312    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda33a7e6_3567_4d5e_a23f_a7540943834a.slice/crio-6f12b111950c7ab5bf6505c0c5dc32923b219092a4b006ebbd61712c90a3252d WatchSource:0}: Error finding container 6f12b111950c7ab5bf6505c0c5dc32923b219092a4b006ebbd61712c90a3252d: Status 404 returned error can't find the container with id 6f12b111950c7ab5bf6505c0c5dc32923b219092a4b006ebbd61712c90a3252d
Jul 27 16:19:11.482734 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:11.482697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"6f12b111950c7ab5bf6505c0c5dc32923b219092a4b006ebbd61712c90a3252d"}
Jul 27 16:19:12.489265 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.489215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42" event={"ID":"f71e5436-f3aa-4a52-ac4d-c91319a29420","Type":"ContainerStarted","Data":"8d71112ea59566111528d591ec101cf66c20868317d6f6daffe15a030abde918"}
Jul 27 16:19:12.489780 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.489324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42" event={"ID":"f71e5436-f3aa-4a52-ac4d-c91319a29420","Type":"ContainerStarted","Data":"2fd39ee029799b9a1a2156e615f8e140ec05ab192a6bddbe3ddc171363843e3e"}
Jul 27 16:19:12.489780 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.489351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42" event={"ID":"f71e5436-f3aa-4a52-ac4d-c91319a29420","Type":"ContainerStarted","Data":"6ad6b48e338c858f83c384a7560e612dbdbad9a1f5620027cc1c4a391e7e7844"}
Jul 27 16:19:12.490829 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.490806    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc" exitCode=0
Jul 27 16:19:12.490917 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.490889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"}
Jul 27 16:19:12.546583 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.546521    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55bdd94856-rvs42" podStartSLOduration=2.285035143 podStartE2EDuration="4.546502856s" totalImagesPullingTime="2.261467713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:12.545354596 +0000 UTC m=+130.170535550" watchObservedRunningTime="2026-07-27 16:19:12.546502856 +0000 UTC m=+130.171683808"
Jul 27 16:19:12.579279 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.579238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:19:12.582080 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.582054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba3bcb10-939b-47af-aad5-da8823db14d1-metrics-certs\") pod \"network-metrics-daemon-vzc6r\" (UID: \"ba3bcb10-939b-47af-aad5-da8823db14d1\") " pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:19:12.878431 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.878337    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9d9gv\""
Jul 27 16:19:12.886597 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:12.886567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vzc6r"
Jul 27 16:19:13.034889 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:13.034855    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vzc6r"]
Jul 27 16:19:13.040181 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:19:13.040144    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3bcb10_939b_47af_aad5_da8823db14d1.slice/crio-8f118c8b06f8b3548c9e1c22f1d52c25323c4f204a88a8dae4b9f279106c9524 WatchSource:0}: Error finding container 8f118c8b06f8b3548c9e1c22f1d52c25323c4f204a88a8dae4b9f279106c9524: Status 404 returned error can't find the container with id 8f118c8b06f8b3548c9e1c22f1d52c25323c4f204a88a8dae4b9f279106c9524
Jul 27 16:19:13.395434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:13.395399    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9fc4f64d-fxjtl"
Jul 27 16:19:13.497358 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:13.497317    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzc6r" event={"ID":"ba3bcb10-939b-47af-aad5-da8823db14d1","Type":"ContainerStarted","Data":"8f118c8b06f8b3548c9e1c22f1d52c25323c4f204a88a8dae4b9f279106c9524"}
Jul 27 16:19:14.503996 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:14.503904    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzc6r" event={"ID":"ba3bcb10-939b-47af-aad5-da8823db14d1","Type":"ContainerStarted","Data":"f2e6e5ed260d6b23041e110c85175d9a7e2798cfdf6b262cd4e7559acc138fec"}
Jul 27 16:19:14.504432 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:14.504037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vzc6r" event={"ID":"ba3bcb10-939b-47af-aad5-da8823db14d1","Type":"ContainerStarted","Data":"38aa39842a0001db187709de6624562c8a1abba5053f13ded515b2799ff00c95"}
Jul 27 16:19:15.043900 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.043848    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84dd7688cc-qp465" podUID="b3355757-a5b4-40b9-8bfb-261753c3cc8d" containerName="registry" containerID="cri-o://6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda" gracePeriod=30
Jul 27 16:19:15.808767 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.808738    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:19:15.842995 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.842937    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vzc6r" podStartSLOduration=132.643514871 podStartE2EDuration="2m13.842916829s" totalImagesPullingTime="1.199401958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:14.528819858 +0000 UTC m=+132.154000812" watchObservedRunningTime="2026-07-27 16:19:15.842916829 +0000 UTC m=+133.468097775"
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.912990    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-bound-sa-token\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.913083    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-installation-pull-secrets\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.913133    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnwbj\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-kube-api-access-lnwbj\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.913167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-image-registry-private-configuration\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.913212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3355757-a5b4-40b9-8bfb-261753c3cc8d-ca-trust-extracted\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.913260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.913609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.913314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-certificates\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.915037 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.914807    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:19:15.915124 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.915042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-trusted-ca\") pod \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\" (UID: \"b3355757-a5b4-40b9-8bfb-261753c3cc8d\") "
Jul 27 16:19:15.915714 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.915349    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-trusted-ca\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:15.916630 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.916356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:19:15.919893 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.919858    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:19:15.920408 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.920343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:19:15.920408 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.920365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:19:15.920688 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.920653    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:19:15.921259 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.921228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-kube-api-access-lnwbj" (OuterVolumeSpecName: "kube-api-access-lnwbj") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "kube-api-access-lnwbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:19:15.927202 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:15.927170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3355757-a5b4-40b9-8bfb-261753c3cc8d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3355757-a5b4-40b9-8bfb-261753c3cc8d" (UID: "b3355757-a5b4-40b9-8bfb-261753c3cc8d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:19:16.016264 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016221    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-image-registry-private-configuration\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.016264 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016255    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3355757-a5b4-40b9-8bfb-261753c3cc8d-ca-trust-extracted\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.016264 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016270    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-tls\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.016571 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016284    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3355757-a5b4-40b9-8bfb-261753c3cc8d-registry-certificates\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.016571 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016297    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-bound-sa-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.016571 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016305    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3355757-a5b4-40b9-8bfb-261753c3cc8d-installation-pull-secrets\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.016571 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.016313    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnwbj\" (UniqueName: \"kubernetes.io/projected/b3355757-a5b4-40b9-8bfb-261753c3cc8d-kube-api-access-lnwbj\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:19:16.515933 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.515874    2579 generic.go:356] "Generic (PLEG): container finished" podID="b3355757-a5b4-40b9-8bfb-261753c3cc8d" containerID="6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda" exitCode=0
Jul 27 16:19:16.516150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.515945    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84dd7688cc-qp465"
Jul 27 16:19:16.516150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.515968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd7688cc-qp465" event={"ID":"b3355757-a5b4-40b9-8bfb-261753c3cc8d","Type":"ContainerDied","Data":"6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda"}
Jul 27 16:19:16.516150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.516051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84dd7688cc-qp465" event={"ID":"b3355757-a5b4-40b9-8bfb-261753c3cc8d","Type":"ContainerDied","Data":"07f7e5bb48644c4d0e88cee857f61b506f04405f8284a2d214bf26888451962a"}
Jul 27 16:19:16.516150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.516085    2579 scope.go:117] "RemoveContainer" containerID="6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda"
Jul 27 16:19:16.520534 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.520493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"}
Jul 27 16:19:16.520667 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.520588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"}
Jul 27 16:19:16.527754 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.527735    2579 scope.go:117] "RemoveContainer" containerID="6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda"
Jul 27 16:19:16.528149 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:19:16.528118    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda\": container with ID starting with 6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda not found: ID does not exist" containerID="6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda"
Jul 27 16:19:16.528265 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.528160    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda"} err="failed to get container status \"6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda\": rpc error: code = NotFound desc = could not find container \"6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda\": container with ID starting with 6c9395b39b303ae67a978f7a08c28bca5319c5608578ecc6df7efdde21594bda not found: ID does not exist"
Jul 27 16:19:16.545580 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.545535    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84dd7688cc-qp465"]
Jul 27 16:19:16.553970 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.553936    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84dd7688cc-qp465"]
Jul 27 16:19:16.968200 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:16.968156    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3355757-a5b4-40b9-8bfb-261753c3cc8d" path="/var/lib/kubelet/pods/b3355757-a5b4-40b9-8bfb-261753c3cc8d/volumes"
Jul 27 16:19:18.537159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:18.537121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"}
Jul 27 16:19:18.537555 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:18.537216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"}
Jul 27 16:19:18.537555 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:18.537242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"}
Jul 27 16:19:18.537555 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:18.537271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerStarted","Data":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"}
Jul 27 16:19:18.572974 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:18.572903    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.362930716 podStartE2EDuration="9.572875546s" totalImagesPullingTime="6.187710324s" totalInitContainerRuntime="22.234506ms" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:18.571547393 +0000 UTC m=+136.196728362" watchObservedRunningTime="2026-07-27 16:19:18.572875546 +0000 UTC m=+136.198056501"
Jul 27 16:19:20.334400 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:20.334340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:19:55.689117 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:55.689070    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b311a84-36bf-4180-bdaa-b100e3f54e11" containerID="efe134d9114300aa30b6a13207d77c8a6666e24b9ad94a8f994e7c2a60f6c417" exitCode=0
Jul 27 16:19:55.689562 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:55.689149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh" event={"ID":"0b311a84-36bf-4180-bdaa-b100e3f54e11","Type":"ContainerDied","Data":"efe134d9114300aa30b6a13207d77c8a6666e24b9ad94a8f994e7c2a60f6c417"}
Jul 27 16:19:55.689562 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:55.689535    2579 scope.go:117] "RemoveContainer" containerID="efe134d9114300aa30b6a13207d77c8a6666e24b9ad94a8f994e7c2a60f6c417"
Jul 27 16:19:56.695526 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:19:56.695492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8xbrh" event={"ID":"0b311a84-36bf-4180-bdaa-b100e3f54e11","Type":"ContainerStarted","Data":"c50eb2c6f19fec9209a552f97eed8edca6512f5cf117ead9bffaf3609c8b9a0a"}
Jul 27 16:20:10.334160 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:10.334114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:10.351742 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:10.351716    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:10.770286 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:10.770255    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.433485 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:20:28.434118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433790    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="prometheus" containerID="cri-o://3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae" gracePeriod=600
Jul 27 16:20:28.434118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433833    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08" gracePeriod=600
Jul 27 16:20:28.434118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433843    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="config-reloader" containerID="cri-o://9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4" gracePeriod=600
Jul 27 16:20:28.434118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433812    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy" containerID="cri-o://223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b" gracePeriod=600
Jul 27 16:20:28.434118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433884    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-web" containerID="cri-o://60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d" gracePeriod=600
Jul 27 16:20:28.434118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.433856    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="thanos-sidecar" containerID="cri-o://f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1" gracePeriod=600
Jul 27 16:20:28.692514 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.692436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.790218 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-metrics-client-certs\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790218 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790218    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790484 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790240    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-trusted-ca-bundle\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790484 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-config-out\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790484 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-kubelet-serving-ca-bundle\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-grpc-tls\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-tls-assets\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790561    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-thanos-prometheus-http-client-file\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-tls\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790639 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-serving-certs-ca-bundle\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-web-config\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-metrics-client-ca\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-kube-rbac-proxy\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-2\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790841    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-config\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.790882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790876    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cxqm\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-kube-api-access-6cxqm\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.791301 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790925    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-db\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.791301 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790961    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-0\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.791301 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.791016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-1\") pod \"da33a7e6-3567-4d5e-a23f-a7540943834a\" (UID: \"da33a7e6-3567-4d5e-a23f-a7540943834a\") "
Jul 27 16:20:28.791627 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.791473    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.793093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.790768    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.793580 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.791352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.793690 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.791786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.793768 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.793459    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.793834 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.793538    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.793907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.793605    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.793986 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.793679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.794109 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.793984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.794380 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.794043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:20:28.794380 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.794077    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.795147 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.795110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.795343 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.795295    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-config-out" (OuterVolumeSpecName: "config-out") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:20:28.795424 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.795345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-kube-api-access-6cxqm" (OuterVolumeSpecName: "kube-api-access-6cxqm") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "kube-api-access-6cxqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:20:28.795507 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.795355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:28.796072 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.796050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.796664 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.796641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-config" (OuterVolumeSpecName: "config") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.796717 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.796667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:20:28.796847 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.796824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.806464 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.806429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-web-config" (OuterVolumeSpecName: "web-config") pod "da33a7e6-3567-4d5e-a23f-a7540943834a" (UID: "da33a7e6-3567-4d5e-a23f-a7540943834a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:28.825173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825138    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08" exitCode=0
Jul 27 16:20:28.825173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825165    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b" exitCode=0
Jul 27 16:20:28.825173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825173    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d" exitCode=0
Jul 27 16:20:28.825173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825179    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1" exitCode=0
Jul 27 16:20:28.825173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825184    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4" exitCode=0
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825189    2579 generic.go:356] "Generic (PLEG): container finished" podID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae" exitCode=0
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825224    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"}
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"}
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"}
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"}
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825425    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.825448 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"}
Jul 27 16:20:28.825708 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"}
Jul 27 16:20:28.825708 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.825490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"da33a7e6-3567-4d5e-a23f-a7540943834a","Type":"ContainerDied","Data":"6f12b111950c7ab5bf6505c0c5dc32923b219092a4b006ebbd61712c90a3252d"}
Jul 27 16:20:28.834914 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.834894    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.843158 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.843136    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.850776 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.850752    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.854655 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.854628    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:20:28.858997 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.858955    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.860777 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.860758    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:20:28.867033 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.866996    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.875357 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.875336    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.883252 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.883229    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.886039 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.884059    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.886039 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.884128    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"} err="failed to get container status \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist"
Jul 27 16:20:28.886039 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.884158    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.888939 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.888907    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.889103 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.888946    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"} err="failed to get container status \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist"
Jul 27 16:20:28.889103 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.888966    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.889341 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.889312    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.889381 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.889352    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"} err="failed to get container status \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist"
Jul 27 16:20:28.889381 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.889374    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.889633 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.889613    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.889683 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.889638    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"} err="failed to get container status \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist"
Jul 27 16:20:28.889683 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.889654    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.889891 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.889870    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.889929 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.889897    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"} err="failed to get container status \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist"
Jul 27 16:20:28.889929 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.889911    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.890170 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.890154    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.890221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.890173    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"} err="failed to get container status \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist"
Jul 27 16:20:28.890221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.890188    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.890420 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:20:28.890379    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.890522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.890429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"} err="failed to get container status \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist"
Jul 27 16:20:28.890522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.890453    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.890720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.890700    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"} err="failed to get container status \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist"
Jul 27 16:20:28.890767 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.890723    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.891298 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.891225    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"} err="failed to get container status \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist"
Jul 27 16:20:28.891426 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.891299    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.891945 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.891918    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"} err="failed to get container status \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist"
Jul 27 16:20:28.891945 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.891945    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892467    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-tls-assets\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892495    2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892521    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892545    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892609    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-web-config\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892630    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892648    2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-configmap-metrics-client-ca\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892667    2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-kube-rbac-proxy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892689    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892708    2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-config\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892727    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cxqm\" (UniqueName: \"kubernetes.io/projected/da33a7e6-3567-4d5e-a23f-a7540943834a-kube-api-access-6cxqm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892749    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-db\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892769    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892788    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892809    2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-metrics-client-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892829    2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.892846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892846    2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da33a7e6-3567-4d5e-a23f-a7540943834a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.893549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892870    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da33a7e6-3567-4d5e-a23f-a7540943834a-config-out\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.893549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892890    2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/da33a7e6-3567-4d5e-a23f-a7540943834a-secret-grpc-tls\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:20:28.893549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892909    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"} err="failed to get container status \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist"
Jul 27 16:20:28.893549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.892943    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.893549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893322    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"} err="failed to get container status \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist"
Jul 27 16:20:28.893549 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893344    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.893770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893593    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"} err="failed to get container status \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist"
Jul 27 16:20:28.893770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893608    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.893770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893661    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:20:28.893904 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893855    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"} err="failed to get container status \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist"
Jul 27 16:20:28.893904 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893880    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.893992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893981    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-thanos"
Jul 27 16:20:28.894062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.893993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-thanos"
Jul 27 16:20:28.894062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894027    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="prometheus"
Jul 27 16:20:28.894062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894036    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="prometheus"
Jul 27 16:20:28.894062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="init-config-reloader"
Jul 27 16:20:28.894062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894058    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="init-config-reloader"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894072    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="thanos-sidecar"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894078    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="thanos-sidecar"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894094    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894102    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3355757-a5b4-40b9-8bfb-261753c3cc8d" containerName="registry"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894107    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3355757-a5b4-40b9-8bfb-261753c3cc8d" containerName="registry"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894113    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="config-reloader"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894118    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="config-reloader"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894124    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-web"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894129    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-web"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894163    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"} err="failed to get container status \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894178    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894198    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-web"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894206    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy-thanos"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894214    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3355757-a5b4-40b9-8bfb-261753c3cc8d" containerName="registry"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894220    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="config-reloader"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894226    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="kube-rbac-proxy"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894237    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="prometheus"
Jul 27 16:20:28.894242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894244    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" containerName="thanos-sidecar"
Jul 27 16:20:28.894835 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894392    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"} err="failed to get container status \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist"
Jul 27 16:20:28.894835 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894406    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.894835 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894614    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"} err="failed to get container status \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist"
Jul 27 16:20:28.894835 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894636    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.894950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894839    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"} err="failed to get container status \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist"
Jul 27 16:20:28.894950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.894860    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.895118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895097    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"} err="failed to get container status \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist"
Jul 27 16:20:28.895162 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895122    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.895383 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895364    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"} err="failed to get container status \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist"
Jul 27 16:20:28.895446 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895383    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.895634 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895613    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"} err="failed to get container status \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist"
Jul 27 16:20:28.895688 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895636    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.895884 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895862    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"} err="failed to get container status \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist"
Jul 27 16:20:28.895884 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.895883    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.896206 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896190    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"} err="failed to get container status \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist"
Jul 27 16:20:28.896206 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896207    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.896418 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896403    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"} err="failed to get container status \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist"
Jul 27 16:20:28.896481 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896419    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.896662 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896642    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"} err="failed to get container status \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist"
Jul 27 16:20:28.896716 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896663    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.896885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896866    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"} err="failed to get container status \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist"
Jul 27 16:20:28.896959 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.896888    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.897139 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897114    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"} err="failed to get container status \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist"
Jul 27 16:20:28.897139 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897132    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.897326 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897310    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"} err="failed to get container status \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist"
Jul 27 16:20:28.897369 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897326    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.897529 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897511    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"} err="failed to get container status \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist"
Jul 27 16:20:28.897573 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897530    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.897747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897729    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"} err="failed to get container status \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist"
Jul 27 16:20:28.897790 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897747    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.897987 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897967    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"} err="failed to get container status \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist"
Jul 27 16:20:28.898066 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.897988    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.898241 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898223    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"} err="failed to get container status \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist"
Jul 27 16:20:28.898313 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898242    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.898473 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898456    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"} err="failed to get container status \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist"
Jul 27 16:20:28.898521 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898475    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.898696 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898679    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"} err="failed to get container status \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist"
Jul 27 16:20:28.898765 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898698    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.899460 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.898878    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"} err="failed to get container status \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist"
Jul 27 16:20:28.899522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.899464    2579 scope.go:117] "RemoveContainer" containerID="93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"
Jul 27 16:20:28.899708 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.899694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.899788 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.899691    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08"} err="failed to get container status \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": rpc error: code = NotFound desc = could not find container \"93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08\": container with ID starting with 93401ade8b99dd895e283a4169ee0d2ce601fa0e138eae651a8bad1132ac5b08 not found: ID does not exist"
Jul 27 16:20:28.899850 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.899792    2579 scope.go:117] "RemoveContainer" containerID="223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"
Jul 27 16:20:28.900292 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.900261    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b"} err="failed to get container status \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": rpc error: code = NotFound desc = could not find container \"223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b\": container with ID starting with 223f62e26c7ed5b8c258e99781cfff95762b3765d9c0233b0d010c0c6beedb3b not found: ID does not exist"
Jul 27 16:20:28.900292 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.900293    2579 scope.go:117] "RemoveContainer" containerID="60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"
Jul 27 16:20:28.900760 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.900740    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d"} err="failed to get container status \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": rpc error: code = NotFound desc = could not find container \"60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d\": container with ID starting with 60ab6d7a10fbc8bdca9799566b2fe464cca53ee99cdc274e4b02dd9209aa3e1d not found: ID does not exist"
Jul 27 16:20:28.900824 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.900762    2579 scope.go:117] "RemoveContainer" containerID="f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"
Jul 27 16:20:28.901037 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.900995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1"} err="failed to get container status \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": rpc error: code = NotFound desc = could not find container \"f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1\": container with ID starting with f3ca4902828dce6d4f5f463f6169b220ec55e0f2f2c59a5425569eae75b846e1 not found: ID does not exist"
Jul 27 16:20:28.901110 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.901040    2579 scope.go:117] "RemoveContainer" containerID="9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"
Jul 27 16:20:28.901291 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.901267    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4"} err="failed to get container status \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": rpc error: code = NotFound desc = could not find container \"9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4\": container with ID starting with 9b93b601c2b73cb69c1e852be04062048fe3d8a10817d1d33f1f8e91c9cbafa4 not found: ID does not exist"
Jul 27 16:20:28.901341 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.901294    2579 scope.go:117] "RemoveContainer" containerID="3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"
Jul 27 16:20:28.901531 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.901509    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae"} err="failed to get container status \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": rpc error: code = NotFound desc = could not find container \"3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae\": container with ID starting with 3051d45a741a2dd270bc9273a4b0c1c9ca16cb8f80402aea11a0083933b2feae not found: ID does not exist"
Jul 27 16:20:28.901586 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.901533    2579 scope.go:117] "RemoveContainer" containerID="2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"
Jul 27 16:20:28.901789 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.901771    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc"} err="failed to get container status \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": rpc error: code = NotFound desc = could not find container \"2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc\": container with ID starting with 2e983575a1771631f2385400241db399efcbd3e0c775e38a0906c1de6637befc not found: ID does not exist"
Jul 27 16:20:28.902486 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 16:20:28.902486 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 16:20:28.902486 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902436    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 16:20:28.902486 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902438    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 16:20:28.902486 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902461    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 16:20:28.902811 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fd55od0fqc1ge\""
Jul 27 16:20:28.902895 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 16:20:28.903023 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.902983    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zrn5q\""
Jul 27 16:20:28.903159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.903135    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 16:20:28.903159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.903144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 16:20:28.905272 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.905249    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 16:20:28.905272 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.905273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 16:20:28.905550 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.905372    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 16:20:28.905550 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.905382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 16:20:28.905753 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.905735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 16:20:28.908545 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.908506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 16:20:28.909718 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.909698    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 16:20:28.914852 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.914820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:20:28.966572 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.966535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da33a7e6-3567-4d5e-a23f-a7540943834a" path="/var/lib/kubelet/pods/da33a7e6-3567-4d5e-a23f-a7540943834a/volumes"
Jul 27 16:20:28.993488 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993591 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993591 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993648 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993648 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993713 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-config\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993713 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b37404e-bec6-44eb-979d-58213529b64d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993778 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993778 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993844 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993844 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993904 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993904 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b37404e-bec6-44eb-979d-58213529b64d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.993962 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.994086 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.993980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.994086 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.994031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgw2m\" (UniqueName: \"kubernetes.io/projected/5b37404e-bec6-44eb-979d-58213529b64d-kube-api-access-vgw2m\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.994143 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.994121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:28.994177 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:28.994141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.094825 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.094825 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-config\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b37404e-bec6-44eb-979d-58213529b64d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095074 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.094990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095552 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.095522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.095950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.095862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096050 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.095987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096232 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096232 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b37404e-bec6-44eb-979d-58213529b64d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096232 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096232 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgw2m\" (UniqueName: \"kubernetes.io/projected/5b37404e-bec6-44eb-979d-58213529b64d-kube-api-access-vgw2m\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.096447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.096413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.098297 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.098267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.098411 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.098310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.098485 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.098424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-config\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099038 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.098888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099151 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.099073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099151 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.099079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5b37404e-bec6-44eb-979d-58213529b64d-config-out\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099369 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.099345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.099330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099648 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.099626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5b37404e-bec6-44eb-979d-58213529b64d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.099815 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.099787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.100047 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.100024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.100129 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.100073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b37404e-bec6-44eb-979d-58213529b64d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.100845 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.100819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.101168 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.101140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-web-config\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.101570 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.101544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.102223 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.102205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5b37404e-bec6-44eb-979d-58213529b64d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.109566 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.109541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgw2m\" (UniqueName: \"kubernetes.io/projected/5b37404e-bec6-44eb-979d-58213529b64d-kube-api-access-vgw2m\") pod \"prometheus-k8s-0\" (UID: \"5b37404e-bec6-44eb-979d-58213529b64d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.211253 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.211215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:20:29.380812 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.380786    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 16:20:29.382909 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:20:29.382883    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b37404e_bec6_44eb_979d_58213529b64d.slice/crio-fd049005a727e25514ce09a45c4bfe3945a415b9f547319e2c0616eadb1ff57c WatchSource:0}: Error finding container fd049005a727e25514ce09a45c4bfe3945a415b9f547319e2c0616eadb1ff57c: Status 404 returned error can't find the container with id fd049005a727e25514ce09a45c4bfe3945a415b9f547319e2c0616eadb1ff57c
Jul 27 16:20:29.831087 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.830967    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b37404e-bec6-44eb-979d-58213529b64d" containerID="885f1c72be02d8cf980cbace97edab4e37df3a4b6f4084b2a6c34c4b43472129" exitCode=0
Jul 27 16:20:29.831087 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.831059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerDied","Data":"885f1c72be02d8cf980cbace97edab4e37df3a4b6f4084b2a6c34c4b43472129"}
Jul 27 16:20:29.831558 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:29.831129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"fd049005a727e25514ce09a45c4bfe3945a415b9f547319e2c0616eadb1ff57c"}
Jul 27 16:20:30.840121 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.840086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"f8ac6020d9cde3b35168102d000419ffc1c4774efe2763998841fcf27611e255"}
Jul 27 16:20:30.840527 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.840178    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"abf46801f06bce4020848331d56354bc2abfd37766647706e754c852b5353cea"}
Jul 27 16:20:30.840527 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.840206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"68901f7500242263721ad81157630f724ad036d61e4ecff3878f7bafd47e3657"}
Jul 27 16:20:30.840527 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.840231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"7b754cbeffb36aded39ac30402ce02afdddb7b3918d8975106b2e2c8da665190"}
Jul 27 16:20:30.840527 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.840261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"d87c3e65b0b9a972737a1b6496695907aef238572db5675cd4f75ee663dd4851"}
Jul 27 16:20:30.840527 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.840286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5b37404e-bec6-44eb-979d-58213529b64d","Type":"ContainerStarted","Data":"8d7c215d58cb864502060fa5b7026ef6c3f5e587630601df1cef38a9a82e01fd"}
Jul 27 16:20:30.877313 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:30.877234    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.85508808 podStartE2EDuration="2.877214305s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.126225ms" isStatefulPod=false podCreationTimestamp="2026-07-27 16:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:20:30.874858556 +0000 UTC m=+208.500039509" watchObservedRunningTime="2026-07-27 16:20:30.877214305 +0000 UTC m=+208.502395258"
Jul 27 16:20:34.211672 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:20:34.211625    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:21:29.212431 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:21:29.212308    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:21:29.229134 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:21:29.229106    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:21:30.086044 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:21:30.085985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 16:22:02.731755 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:02.731722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:22:02.735071 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:02.735034    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:22:02.743062 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:02.743034    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 16:22:36.854828 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:36.854787    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"]
Jul 27 16:22:36.858165 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:36.858145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:36.866323 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:36.861425    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 16:22:36.866323 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:36.861770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-ppq6b\""
Jul 27 16:22:36.866323 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:36.862044    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 16:22:36.894926 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:36.894893    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"]
Jul 27 16:22:37.037381 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.037336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nwpq\" (UniqueName: \"kubernetes.io/projected/a07be0e9-9d31-46d2-a5e6-9babf4cac035-kube-api-access-4nwpq\") pod \"openshift-lws-operator-fd8ccff4c-jjqtr\" (UID: \"a07be0e9-9d31-46d2-a5e6-9babf4cac035\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.037381 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.037384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07be0e9-9d31-46d2-a5e6-9babf4cac035-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jjqtr\" (UID: \"a07be0e9-9d31-46d2-a5e6-9babf4cac035\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.139042 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.138902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nwpq\" (UniqueName: \"kubernetes.io/projected/a07be0e9-9d31-46d2-a5e6-9babf4cac035-kube-api-access-4nwpq\") pod \"openshift-lws-operator-fd8ccff4c-jjqtr\" (UID: \"a07be0e9-9d31-46d2-a5e6-9babf4cac035\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.139042 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.138948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07be0e9-9d31-46d2-a5e6-9babf4cac035-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jjqtr\" (UID: \"a07be0e9-9d31-46d2-a5e6-9babf4cac035\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.139346 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.139326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a07be0e9-9d31-46d2-a5e6-9babf4cac035-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jjqtr\" (UID: \"a07be0e9-9d31-46d2-a5e6-9babf4cac035\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.149211 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.149174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nwpq\" (UniqueName: \"kubernetes.io/projected/a07be0e9-9d31-46d2-a5e6-9babf4cac035-kube-api-access-4nwpq\") pod \"openshift-lws-operator-fd8ccff4c-jjqtr\" (UID: \"a07be0e9-9d31-46d2-a5e6-9babf4cac035\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.183813 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.183784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"
Jul 27 16:22:37.329157 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.328974    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr"]
Jul 27 16:22:37.332069 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:22:37.332034    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07be0e9_9d31_46d2_a5e6_9babf4cac035.slice/crio-355942a2dd502c9ab570225f6859afe3016c43f3b0d925adc866d81120e0ae10 WatchSource:0}: Error finding container 355942a2dd502c9ab570225f6859afe3016c43f3b0d925adc866d81120e0ae10: Status 404 returned error can't find the container with id 355942a2dd502c9ab570225f6859afe3016c43f3b0d925adc866d81120e0ae10
Jul 27 16:22:37.333742 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:37.333718    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:22:38.334497 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:38.334452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr" event={"ID":"a07be0e9-9d31-46d2-a5e6-9babf4cac035","Type":"ContainerStarted","Data":"355942a2dd502c9ab570225f6859afe3016c43f3b0d925adc866d81120e0ae10"}
Jul 27 16:22:40.344401 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:40.344362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr" event={"ID":"a07be0e9-9d31-46d2-a5e6-9babf4cac035","Type":"ContainerStarted","Data":"faab651ac28a3537a8f66e1d71ec57eed51a96683673926b7490d14bf3ae6072"}
Jul 27 16:22:40.364735 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:40.364669    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jjqtr" podStartSLOduration=2.070071825 podStartE2EDuration="4.364651136s" totalImagesPullingTime="2.294579311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:22:40.36411612 +0000 UTC m=+337.989297073" watchObservedRunningTime="2026-07-27 16:22:40.364651136 +0000 UTC m=+337.989832089"
Jul 27 16:22:59.209208 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.209122    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"]
Jul 27 16:22:59.212648 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.212613    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.219736 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.219491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 16:22:59.219876 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.219854    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 16:22:59.220163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.220139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 16:22:59.220163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.220157    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 16:22:59.220936 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.220905    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jgz5w\""
Jul 27 16:22:59.236712 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.236681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"]
Jul 27 16:22:59.336288 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.336242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/09b45e37-3846-4ea5-be60-cdf5b6e0f458-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.336288 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.336289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b45e37-3846-4ea5-be60-cdf5b6e0f458-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.336509 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.336311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxg4\" (UniqueName: \"kubernetes.io/projected/09b45e37-3846-4ea5-be60-cdf5b6e0f458-kube-api-access-5jxg4\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.336509 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.336362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/09b45e37-3846-4ea5-be60-cdf5b6e0f458-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.336509 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.336411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09b45e37-3846-4ea5-be60-cdf5b6e0f458-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.437735 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.437682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/09b45e37-3846-4ea5-be60-cdf5b6e0f458-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.437735 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.437736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b45e37-3846-4ea5-be60-cdf5b6e0f458-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.438032 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.437762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxg4\" (UniqueName: \"kubernetes.io/projected/09b45e37-3846-4ea5-be60-cdf5b6e0f458-kube-api-access-5jxg4\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.438032 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.437793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/09b45e37-3846-4ea5-be60-cdf5b6e0f458-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.438032 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.437823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09b45e37-3846-4ea5-be60-cdf5b6e0f458-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.440043 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.438718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/09b45e37-3846-4ea5-be60-cdf5b6e0f458-manifests\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.441456 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.441423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/09b45e37-3846-4ea5-be60-cdf5b6e0f458-tmp\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.441669 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.441646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/09b45e37-3846-4ea5-be60-cdf5b6e0f458-webhook-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.447267 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.447232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxg4\" (UniqueName: \"kubernetes.io/projected/09b45e37-3846-4ea5-be60-cdf5b6e0f458-kube-api-access-5jxg4\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.447415 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.447382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/09b45e37-3846-4ea5-be60-cdf5b6e0f458-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55548c94b5-bz8jd\" (UID: \"09b45e37-3846-4ea5-be60-cdf5b6e0f458\") " pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.523433 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.523332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:22:59.677120 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:22:59.677084    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"]
Jul 27 16:22:59.680692 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:22:59.680661    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b45e37_3846_4ea5_be60_cdf5b6e0f458.slice/crio-b03041b88e916c6d4a91a69b993f382aa6e0c26157848a4fa172ba6dfd79ac79 WatchSource:0}: Error finding container b03041b88e916c6d4a91a69b993f382aa6e0c26157848a4fa172ba6dfd79ac79: Status 404 returned error can't find the container with id b03041b88e916c6d4a91a69b993f382aa6e0c26157848a4fa172ba6dfd79ac79
Jul 27 16:23:00.429944 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:00.429908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd" event={"ID":"09b45e37-3846-4ea5-be60-cdf5b6e0f458","Type":"ContainerStarted","Data":"b03041b88e916c6d4a91a69b993f382aa6e0c26157848a4fa172ba6dfd79ac79"}
Jul 27 16:23:03.450087 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:03.450052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd" event={"ID":"09b45e37-3846-4ea5-be60-cdf5b6e0f458","Type":"ContainerStarted","Data":"4e35a0edbf9b5c4a477666366de03a346f25d258bd30a6e3820bbeed742aa9cb"}
Jul 27 16:23:04.456318 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:04.456279    2579 generic.go:356] "Generic (PLEG): container finished" podID="09b45e37-3846-4ea5-be60-cdf5b6e0f458" containerID="4e35a0edbf9b5c4a477666366de03a346f25d258bd30a6e3820bbeed742aa9cb" exitCode=0
Jul 27 16:23:04.456318 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:04.456321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd" event={"ID":"09b45e37-3846-4ea5-be60-cdf5b6e0f458","Type":"ContainerDied","Data":"4e35a0edbf9b5c4a477666366de03a346f25d258bd30a6e3820bbeed742aa9cb"}
Jul 27 16:23:05.462768 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:05.462734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd" event={"ID":"09b45e37-3846-4ea5-be60-cdf5b6e0f458","Type":"ContainerStarted","Data":"1e06a7d2ab6b69ce029f63f9662649e50f8c658f8362586c0f89796bec5fcc06"}
Jul 27 16:23:05.463204 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:05.462866    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:23:05.492059 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:05.491976    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd" podStartSLOduration=1.436665145 podStartE2EDuration="6.491957279s" totalImagesPullingTime="3.966531281s" totalInitContainerRuntime="1.088760853s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:22:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:05.491263282 +0000 UTC m=+363.116444235" watchObservedRunningTime="2026-07-27 16:23:05.491957279 +0000 UTC m=+363.117138232"
Jul 27 16:23:16.468865 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:16.468834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-55548c94b5-bz8jd"
Jul 27 16:23:21.950019 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:21.949965    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"]
Jul 27 16:23:21.959992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:21.959964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:21.964665 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:21.964638    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 16:23:21.965979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:21.965769    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 16:23:21.965979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:21.965773    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9mpv9\""
Jul 27 16:23:21.968525 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:21.968502    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"]
Jul 27 16:23:22.038984 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.038947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b4199b8-d637-495d-95ba-a680a2e4c59c-tmp\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.039221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.038991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4199b8-d637-495d-95ba-a680a2e4c59c-tls-certs\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.039221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.039069    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6qw\" (UniqueName: \"kubernetes.io/projected/9b4199b8-d637-495d-95ba-a680a2e4c59c-kube-api-access-gq6qw\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.140343 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.140304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b4199b8-d637-495d-95ba-a680a2e4c59c-tmp\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.140557 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.140352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4199b8-d637-495d-95ba-a680a2e4c59c-tls-certs\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.140557 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.140392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6qw\" (UniqueName: \"kubernetes.io/projected/9b4199b8-d637-495d-95ba-a680a2e4c59c-kube-api-access-gq6qw\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.142882 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.142855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b4199b8-d637-495d-95ba-a680a2e4c59c-tmp\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.143236 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.143198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b4199b8-d637-495d-95ba-a680a2e4c59c-tls-certs\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.151768 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.151735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6qw\" (UniqueName: \"kubernetes.io/projected/9b4199b8-d637-495d-95ba-a680a2e4c59c-kube-api-access-gq6qw\") pod \"kube-auth-proxy-78dbfdf94f-4cj5x\" (UID: \"9b4199b8-d637-495d-95ba-a680a2e4c59c\") " pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.270648 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.270556    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"
Jul 27 16:23:22.425629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.425572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x"]
Jul 27 16:23:22.428370 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:23:22.428337    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4199b8_d637_495d_95ba_a680a2e4c59c.slice/crio-db83daa101556e8c0fd4469fa60d8fa341c991737609b46e3d9beb2bebf30d95 WatchSource:0}: Error finding container db83daa101556e8c0fd4469fa60d8fa341c991737609b46e3d9beb2bebf30d95: Status 404 returned error can't find the container with id db83daa101556e8c0fd4469fa60d8fa341c991737609b46e3d9beb2bebf30d95
Jul 27 16:23:22.540707 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:22.540608    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x" event={"ID":"9b4199b8-d637-495d-95ba-a680a2e4c59c","Type":"ContainerStarted","Data":"db83daa101556e8c0fd4469fa60d8fa341c991737609b46e3d9beb2bebf30d95"}
Jul 27 16:23:24.583453 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.583411    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"]
Jul 27 16:23:24.587042 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.586994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.590827 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.590800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 16:23:24.591028 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.590864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 16:23:24.591028 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.590890    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4gzbz\""
Jul 27 16:23:24.591028 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.590859    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 16:23:24.597246 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.597217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"]
Jul 27 16:23:24.665844 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.665806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-metrics-cert\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.666050 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.665869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57nb\" (UniqueName: \"kubernetes.io/projected/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-kube-api-access-x57nb\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.666050 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.665970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-cert\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.666172 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.666048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-manager-config\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.767244 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.767206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-manager-config\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.767442 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.767313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-metrics-cert\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.767442 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.767353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57nb\" (UniqueName: \"kubernetes.io/projected/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-kube-api-access-x57nb\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.767442 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.767408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-cert\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.768113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.768062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-manager-config\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.771025 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.770962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-metrics-cert\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.771973 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.771940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-cert\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.777212 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.777185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57nb\" (UniqueName: \"kubernetes.io/projected/a6c8b3e9-05f8-41ae-8de7-4baf56d10838-kube-api-access-x57nb\") pod \"lws-controller-manager-cd69fdcb8-g8c66\" (UID: \"a6c8b3e9-05f8-41ae-8de7-4baf56d10838\") " pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:24.898818 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:24.898720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:25.394996 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:25.394969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"]
Jul 27 16:23:25.397731 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:23:25.397703    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c8b3e9_05f8_41ae_8de7_4baf56d10838.slice/crio-90184689661ec6a0a7df95eb740fe904dd92c8a351e065b5b10515a302749d26 WatchSource:0}: Error finding container 90184689661ec6a0a7df95eb740fe904dd92c8a351e065b5b10515a302749d26: Status 404 returned error can't find the container with id 90184689661ec6a0a7df95eb740fe904dd92c8a351e065b5b10515a302749d26
Jul 27 16:23:25.555350 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:25.555234    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66" event={"ID":"a6c8b3e9-05f8-41ae-8de7-4baf56d10838","Type":"ContainerStarted","Data":"90184689661ec6a0a7df95eb740fe904dd92c8a351e065b5b10515a302749d26"}
Jul 27 16:23:25.556656 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:25.556626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x" event={"ID":"9b4199b8-d637-495d-95ba-a680a2e4c59c","Type":"ContainerStarted","Data":"182684bd92849bc8ab6ca38e5225ecacf78f16e2a52c2c68598ae332ae149c2f"}
Jul 27 16:23:25.575821 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:25.575766    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78dbfdf94f-4cj5x" podStartSLOduration=1.705653477 podStartE2EDuration="4.575748196s" totalImagesPullingTime="2.870094719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:25.574905303 +0000 UTC m=+383.200086256" watchObservedRunningTime="2026-07-27 16:23:25.575748196 +0000 UTC m=+383.200929149"
Jul 27 16:23:27.569785 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:27.569751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66" event={"ID":"a6c8b3e9-05f8-41ae-8de7-4baf56d10838","Type":"ContainerStarted","Data":"c71dd07a1b4deb211928d4761056967000008f4b51d1e2d7e54c249d17a5c2f9"}
Jul 27 16:23:27.570242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:27.569884    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:23:27.599307 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:27.599250    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66" podStartSLOduration=2.018363042 podStartE2EDuration="3.599227599s" totalImagesPullingTime="1.580864557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:27.598918079 +0000 UTC m=+385.224099036" watchObservedRunningTime="2026-07-27 16:23:27.599227599 +0000 UTC m=+385.224408553"
Jul 27 16:23:38.577099 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:23:38.577067    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-cd69fdcb8-g8c66"
Jul 27 16:25:17.242222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.242173    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"]
Jul 27 16:25:17.245779 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.245752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.249609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.249571    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 16:25:17.249609 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.249586    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zjx4c\""
Jul 27 16:25:17.250017 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.249988    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 16:25:17.267765 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.267729    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"]
Jul 27 16:25:17.377658 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.377599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca51899d-e12b-46cf-bf38-c249ce056ea6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7m5fn\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.377886 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.377674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6pdt\" (UniqueName: \"kubernetes.io/projected/ca51899d-e12b-46cf-bf38-c249ce056ea6-kube-api-access-j6pdt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7m5fn\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.479192 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.479142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca51899d-e12b-46cf-bf38-c249ce056ea6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7m5fn\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.479420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.479206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6pdt\" (UniqueName: \"kubernetes.io/projected/ca51899d-e12b-46cf-bf38-c249ce056ea6-kube-api-access-j6pdt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7m5fn\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.479591 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.479567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca51899d-e12b-46cf-bf38-c249ce056ea6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7m5fn\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.492878 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.492798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6pdt\" (UniqueName: \"kubernetes.io/projected/ca51899d-e12b-46cf-bf38-c249ce056ea6-kube-api-access-j6pdt\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7m5fn\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.558212 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.558157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:17.723118 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:17.722948    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"]
Jul 27 16:25:17.726288 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:25:17.726257    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca51899d_e12b_46cf_bf38_c249ce056ea6.slice/crio-bcf9eafcd9fb74bb5ecb08195a7897a4987d3b8df24394bebdba808163fbcaf5 WatchSource:0}: Error finding container bcf9eafcd9fb74bb5ecb08195a7897a4987d3b8df24394bebdba808163fbcaf5: Status 404 returned error can't find the container with id bcf9eafcd9fb74bb5ecb08195a7897a4987d3b8df24394bebdba808163fbcaf5
Jul 27 16:25:18.047895 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:18.047857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn" event={"ID":"ca51899d-e12b-46cf-bf38-c249ce056ea6","Type":"ContainerStarted","Data":"bcf9eafcd9fb74bb5ecb08195a7897a4987d3b8df24394bebdba808163fbcaf5"}
Jul 27 16:25:23.075619 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:23.075512    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn" event={"ID":"ca51899d-e12b-46cf-bf38-c249ce056ea6","Type":"ContainerStarted","Data":"e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366"}
Jul 27 16:25:23.076291 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:23.075645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:23.103270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:23.103209    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn" podStartSLOduration=1.103190225 podStartE2EDuration="6.103191222s" totalImagesPullingTime="5.000000997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:23.101706521 +0000 UTC m=+500.726887474" watchObservedRunningTime="2026-07-27 16:25:23.103191222 +0000 UTC m=+500.728372174"
Jul 27 16:25:29.427175 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.427134    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"]
Jul 27 16:25:29.427642 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.427320    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn" podUID="ca51899d-e12b-46cf-bf38-c249ce056ea6" containerName="manager" containerID="cri-o://e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366" gracePeriod=10
Jul 27 16:25:29.429572 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.429544    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:29.682611 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.682530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:29.803931 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.803893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca51899d-e12b-46cf-bf38-c249ce056ea6-extensions-socket-volume\") pod \"ca51899d-e12b-46cf-bf38-c249ce056ea6\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") "
Jul 27 16:25:29.804136 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.803963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6pdt\" (UniqueName: \"kubernetes.io/projected/ca51899d-e12b-46cf-bf38-c249ce056ea6-kube-api-access-j6pdt\") pod \"ca51899d-e12b-46cf-bf38-c249ce056ea6\" (UID: \"ca51899d-e12b-46cf-bf38-c249ce056ea6\") "
Jul 27 16:25:29.804487 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.804465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca51899d-e12b-46cf-bf38-c249ce056ea6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ca51899d-e12b-46cf-bf38-c249ce056ea6" (UID: "ca51899d-e12b-46cf-bf38-c249ce056ea6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:25:29.806427 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.806404    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca51899d-e12b-46cf-bf38-c249ce056ea6-kube-api-access-j6pdt" (OuterVolumeSpecName: "kube-api-access-j6pdt") pod "ca51899d-e12b-46cf-bf38-c249ce056ea6" (UID: "ca51899d-e12b-46cf-bf38-c249ce056ea6"). InnerVolumeSpecName "kube-api-access-j6pdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:25:29.905666 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.905621    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6pdt\" (UniqueName: \"kubernetes.io/projected/ca51899d-e12b-46cf-bf38-c249ce056ea6-kube-api-access-j6pdt\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:25:29.905666 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:29.905653    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca51899d-e12b-46cf-bf38-c249ce056ea6-extensions-socket-volume\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:25:30.109161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.109118    2579 generic.go:356] "Generic (PLEG): container finished" podID="ca51899d-e12b-46cf-bf38-c249ce056ea6" containerID="e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366" exitCode=0
Jul 27 16:25:30.109161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.109164    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"
Jul 27 16:25:30.109430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.109203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn" event={"ID":"ca51899d-e12b-46cf-bf38-c249ce056ea6","Type":"ContainerDied","Data":"e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366"}
Jul 27 16:25:30.109430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.109266    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn" event={"ID":"ca51899d-e12b-46cf-bf38-c249ce056ea6","Type":"ContainerDied","Data":"bcf9eafcd9fb74bb5ecb08195a7897a4987d3b8df24394bebdba808163fbcaf5"}
Jul 27 16:25:30.109430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.109300    2579 scope.go:117] "RemoveContainer" containerID="e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366"
Jul 27 16:25:30.119792 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.119764    2579 scope.go:117] "RemoveContainer" containerID="e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366"
Jul 27 16:25:30.120246 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:25:30.120223    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366\": container with ID starting with e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366 not found: ID does not exist" containerID="e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366"
Jul 27 16:25:30.120341 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.120255    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366"} err="failed to get container status \"e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366\": rpc error: code = NotFound desc = could not find container \"e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366\": container with ID starting with e74b8998555ba725d7e6a17d709bac23f2f4da0a9698a3acf688e16fa7220366 not found: ID does not exist"
Jul 27 16:25:30.141790 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.141757    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"]
Jul 27 16:25:30.146099 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.146060    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7m5fn"]
Jul 27 16:25:30.967151 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:30.967118    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca51899d-e12b-46cf-bf38-c249ce056ea6" path="/var/lib/kubelet/pods/ca51899d-e12b-46cf-bf38-c249ce056ea6/volumes"
Jul 27 16:25:45.798538 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.798500    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"]
Jul 27 16:25:45.798929 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.798826    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca51899d-e12b-46cf-bf38-c249ce056ea6" containerName="manager"
Jul 27 16:25:45.798929 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.798837    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca51899d-e12b-46cf-bf38-c249ce056ea6" containerName="manager"
Jul 27 16:25:45.798929 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.798903    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca51899d-e12b-46cf-bf38-c249ce056ea6" containerName="manager"
Jul 27 16:25:45.803328 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.803303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.806364 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.806336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srw56\""
Jul 27 16:25:45.806696 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.806677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 16:25:45.821502 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.821465    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"]
Jul 27 16:25:45.837862 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.837822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838069 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.837869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838069 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.837949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838069 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.838030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838069 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.838064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838227 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.838111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838227 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.838162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838227 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.838190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.838328 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.838231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzsm4\" (UniqueName: \"kubernetes.io/projected/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-kube-api-access-pzsm4\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939173 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzsm4\" (UniqueName: \"kubernetes.io/projected/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-kube-api-access-pzsm4\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939812 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939812 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939812 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.939965 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.939937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.940170 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.940145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.941995 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.941972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.942229 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.942211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.948976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.948946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:45.949204 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:45.949184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzsm4\" (UniqueName: \"kubernetes.io/projected/1ecfd50d-aadb-4df8-9249-25f31fe1cda9-kube-api-access-pzsm4\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qkq6j\" (UID: \"1ecfd50d-aadb-4df8-9249-25f31fe1cda9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:46.114627 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:46.114521    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:46.292180 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:46.292137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"]
Jul 27 16:25:46.294855 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:25:46.294824    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecfd50d_aadb_4df8_9249_25f31fe1cda9.slice/crio-c701a79fe928d1e483dd6794198f89b4f37ec7331dca80f9139e0960890b321e WatchSource:0}: Error finding container c701a79fe928d1e483dd6794198f89b4f37ec7331dca80f9139e0960890b321e: Status 404 returned error can't find the container with id c701a79fe928d1e483dd6794198f89b4f37ec7331dca80f9139e0960890b321e
Jul 27 16:25:47.190492 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:47.190452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" event={"ID":"1ecfd50d-aadb-4df8-9249-25f31fe1cda9","Type":"ContainerStarted","Data":"c701a79fe928d1e483dd6794198f89b4f37ec7331dca80f9139e0960890b321e"}
Jul 27 16:25:49.177868 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:49.177830    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:25:49.178207 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:49.177904    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:25:49.178207 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:49.177936    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:25:50.207674 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:50.207632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" event={"ID":"1ecfd50d-aadb-4df8-9249-25f31fe1cda9","Type":"ContainerStarted","Data":"440284647b915a857d524a3987b625c35c928e9ae2bd2c4106c2915c8b534561"}
Jul 27 16:25:50.230953 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:50.230897    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" podStartSLOduration=2.350519559 podStartE2EDuration="5.230879537s" totalImagesPullingTime="2.880359978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:50.229425682 +0000 UTC m=+527.854606636" watchObservedRunningTime="2026-07-27 16:25:50.230879537 +0000 UTC m=+527.856060489"
Jul 27 16:25:51.114710 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:51.114671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:51.119667 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:51.119637    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:51.211851 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:51.211816    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:25:51.213055 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:25:51.213034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:26:05.181910 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.181866    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:26:05.186868 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.186845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.190467 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.190439    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 16:26:05.190747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.190729    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 16:26:05.191528 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.191511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2xkwb\""
Jul 27 16:26:05.191647 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.191541    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 16:26:05.197628 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.197598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:26:05.228339 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.228303    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:26:05.310729 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.310684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ae6a93c-61e4-4d8b-a493-de25d5d31240-config-file\") pod \"limitador-limitador-54d6d988d5-x2l82\" (UID: \"5ae6a93c-61e4-4d8b-a493-de25d5d31240\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.310919 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.310756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ptvw\" (UniqueName: \"kubernetes.io/projected/5ae6a93c-61e4-4d8b-a493-de25d5d31240-kube-api-access-4ptvw\") pod \"limitador-limitador-54d6d988d5-x2l82\" (UID: \"5ae6a93c-61e4-4d8b-a493-de25d5d31240\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.411653 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.411607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ae6a93c-61e4-4d8b-a493-de25d5d31240-config-file\") pod \"limitador-limitador-54d6d988d5-x2l82\" (UID: \"5ae6a93c-61e4-4d8b-a493-de25d5d31240\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.411869 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.411686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ptvw\" (UniqueName: \"kubernetes.io/projected/5ae6a93c-61e4-4d8b-a493-de25d5d31240-kube-api-access-4ptvw\") pod \"limitador-limitador-54d6d988d5-x2l82\" (UID: \"5ae6a93c-61e4-4d8b-a493-de25d5d31240\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.412473 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.412432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5ae6a93c-61e4-4d8b-a493-de25d5d31240-config-file\") pod \"limitador-limitador-54d6d988d5-x2l82\" (UID: \"5ae6a93c-61e4-4d8b-a493-de25d5d31240\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.421400 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.421374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ptvw\" (UniqueName: \"kubernetes.io/projected/5ae6a93c-61e4-4d8b-a493-de25d5d31240-kube-api-access-4ptvw\") pod \"limitador-limitador-54d6d988d5-x2l82\" (UID: \"5ae6a93c-61e4-4d8b-a493-de25d5d31240\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.499320 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.499278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:05.655617 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:05.655589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:26:05.658354 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:26:05.658308    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae6a93c_61e4_4d8b_a493_de25d5d31240.slice/crio-e4263678741f4457f3786faa7c3c767b1f97ec48bda9dae860accc3d76cdfd8e WatchSource:0}: Error finding container e4263678741f4457f3786faa7c3c767b1f97ec48bda9dae860accc3d76cdfd8e: Status 404 returned error can't find the container with id e4263678741f4457f3786faa7c3c767b1f97ec48bda9dae860accc3d76cdfd8e
Jul 27 16:26:06.290846 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:06.290800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82" event={"ID":"5ae6a93c-61e4-4d8b-a493-de25d5d31240","Type":"ContainerStarted","Data":"e4263678741f4457f3786faa7c3c767b1f97ec48bda9dae860accc3d76cdfd8e"}
Jul 27 16:26:09.308954 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:09.308918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82" event={"ID":"5ae6a93c-61e4-4d8b-a493-de25d5d31240","Type":"ContainerStarted","Data":"140b652ead138fddd1f508369db252e9b59d426f6d893b90f4ff868d9104d3ee"}
Jul 27 16:26:09.309368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:09.309057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:26:09.333416 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:09.333357    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82" podStartSLOduration=1.6171134120000001 podStartE2EDuration="4.33334039s" totalImagesPullingTime="2.716226978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:26:09.332336139 +0000 UTC m=+546.957517119" watchObservedRunningTime="2026-07-27 16:26:09.33334039 +0000 UTC m=+546.958521342"
Jul 27 16:26:20.314547 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:26:20.314513    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-x2l82"
Jul 27 16:27:02.763841 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:02.763806    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:27:02.764463 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:02.764430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:27:18.869524 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:18.869478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:27:26.662719 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:26.662683    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ecfd50d-aadb-4df8-9249-25f31fe1cda9" containerID="440284647b915a857d524a3987b625c35c928e9ae2bd2c4106c2915c8b534561" exitCode=0
Jul 27 16:27:26.663174 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:26.662741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" event={"ID":"1ecfd50d-aadb-4df8-9249-25f31fe1cda9","Type":"ContainerDied","Data":"440284647b915a857d524a3987b625c35c928e9ae2bd2c4106c2915c8b534561"}
Jul 27 16:27:26.663220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:26.663197    2579 scope.go:117] "RemoveContainer" containerID="440284647b915a857d524a3987b625c35c928e9ae2bd2c4106c2915c8b534561"
Jul 27 16:27:26.664329 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:26.664294    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:27:26.664396 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:26.664376    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:27:26.664434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:26.664415    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:27:27.669522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:27.669489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" event={"ID":"1ecfd50d-aadb-4df8-9249-25f31fe1cda9","Type":"ContainerStarted","Data":"fc19b28594414181cf64100d7b8ce6c951a20332c773d9b0cb73aa9c5bccdfd6"}
Jul 27 16:27:28.115831 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:28.115791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:27:28.116293 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:28.116185    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qkq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 27 16:27:28.116511 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:28.116478    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" podUID="1ecfd50d-aadb-4df8-9249-25f31fe1cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 27 16:27:29.115029 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:29.114967    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qkq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 27 16:27:29.115663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:29.115627    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" podUID="1ecfd50d-aadb-4df8-9249-25f31fe1cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 27 16:27:30.115384 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:30.115337    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qkq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 27 16:27:30.116043 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:30.115982    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" podUID="1ecfd50d-aadb-4df8-9249-25f31fe1cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 27 16:27:31.115289 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:31.115243    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qkq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body=
Jul 27 16:27:31.115619 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:31.115581    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j" podUID="1ecfd50d-aadb-4df8-9249-25f31fe1cda9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused"
Jul 27 16:27:32.119348 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:32.119317    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:27:32.119797 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:32.119603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:27:32.120546 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:32.120523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qkq6j"
Jul 27 16:27:39.659491 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:39.659450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:27:44.260186 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:44.260148    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:27:48.357504 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:27:48.357469    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:28:08.674759 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:08.674722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:28:27.224926 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.224890    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"]
Jul 27 16:28:27.228364 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.228344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.232265 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.232234    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 16:28:27.233396 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.233370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 16:28:27.233519 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.233370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tgshm\""
Jul 27 16:28:27.233519 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.233372    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 27 16:28:27.259979 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.259943    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"]
Jul 27 16:28:27.334516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.334471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f488\" (UniqueName: \"kubernetes.io/projected/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-kube-api-access-5f488\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.334671 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.334563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.334671 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.334590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.334766 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.334673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.334766 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.334733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.334766 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.334760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.435727 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.435684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.435727 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.435730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436036 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.435765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f488\" (UniqueName: \"kubernetes.io/projected/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-kube-api-access-5f488\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436036 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.435812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436036 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.435831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436036 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.435871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436261 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.436206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436261 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.436221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.436261 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.436250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.438567 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.438540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.438746 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.438728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.446608 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.446574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f488\" (UniqueName: \"kubernetes.io/projected/f3427f5f-fba3-415a-bf6a-c3f0637ffa38-kube-api-access-5f488\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-f8w9b\" (UID: \"f3427f5f-fba3-415a-bf6a-c3f0637ffa38\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.538723 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.538620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:27.719590 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.719564    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"]
Jul 27 16:28:27.722286 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:28:27.722248    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3427f5f_fba3_415a_bf6a_c3f0637ffa38.slice/crio-79e050210ecb65873d1949d18b47dfc415c895e575cae6af55e85c5760ac54ce WatchSource:0}: Error finding container 79e050210ecb65873d1949d18b47dfc415c895e575cae6af55e85c5760ac54ce: Status 404 returned error can't find the container with id 79e050210ecb65873d1949d18b47dfc415c895e575cae6af55e85c5760ac54ce
Jul 27 16:28:27.723795 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.723770    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:28:27.945159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:27.945053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b" event={"ID":"f3427f5f-fba3-415a-bf6a-c3f0637ffa38","Type":"ContainerStarted","Data":"79e050210ecb65873d1949d18b47dfc415c895e575cae6af55e85c5760ac54ce"}
Jul 27 16:28:28.952564 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:28.952519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b" event={"ID":"f3427f5f-fba3-415a-bf6a-c3f0637ffa38","Type":"ContainerStarted","Data":"c066f3137ac56aeb79f800ab9b2c70adefe43691978da604d2cdb3d4b8c2f801"}
Jul 27 16:28:28.974239 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:28.974188    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b" podStartSLOduration=0.934012022 podStartE2EDuration="1.974171406s" totalImagesPullingTime="1.040159384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:28:28.973734779 +0000 UTC m=+686.598915731" watchObservedRunningTime="2026-07-27 16:28:28.974171406 +0000 UTC m=+686.599352358"
Jul 27 16:28:30.168546 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:30.168500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:28:37.539115 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:37.539062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:37.539554 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:37.539143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:37.552259 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:37.552227    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:38.008559 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:38.008527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-f8w9b"
Jul 27 16:28:48.754818 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.754779    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"]
Jul 27 16:28:48.758585 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.758564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.762209 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.762185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 16:28:48.772134 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.772097    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"]
Jul 27 16:28:48.838332 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.838289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.838550 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.838348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.838550 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.838416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czwl8\" (UniqueName: \"kubernetes.io/projected/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-kube-api-access-czwl8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.838550 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.838478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.838550 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.838520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.838723 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.838579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.939878 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.939836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czwl8\" (UniqueName: \"kubernetes.io/projected/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-kube-api-access-czwl8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.939900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.939929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.939972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.940023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940098 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.940079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940414 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.940390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940474 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.940424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.940519 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.940482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.942533 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.942509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.942715 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.942697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:48.951206 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:48.951173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czwl8\" (UniqueName: \"kubernetes.io/projected/5b0fe6ff-a7ff-452d-91f9-9861471e2d99-kube-api-access-czwl8\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp\" (UID: \"5b0fe6ff-a7ff-452d-91f9-9861471e2d99\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:49.068526 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:49.068408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:49.216771 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:49.216745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"]
Jul 27 16:28:49.218418 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:28:49.218390    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b0fe6ff_a7ff_452d_91f9_9861471e2d99.slice/crio-f36360a5712cc7286901282aedf61afa87cf68493995907bf1a908270bae7c11 WatchSource:0}: Error finding container f36360a5712cc7286901282aedf61afa87cf68493995907bf1a908270bae7c11: Status 404 returned error can't find the container with id f36360a5712cc7286901282aedf61afa87cf68493995907bf1a908270bae7c11
Jul 27 16:28:50.052802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:50.052768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp" event={"ID":"5b0fe6ff-a7ff-452d-91f9-9861471e2d99","Type":"ContainerStarted","Data":"b35a4d9104304fdbf26969fbfbbe6642c5855bb99bdfbe375d0fab3dacaeb4f9"}
Jul 27 16:28:50.052802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:50.052808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp" event={"ID":"5b0fe6ff-a7ff-452d-91f9-9861471e2d99","Type":"ContainerStarted","Data":"f36360a5712cc7286901282aedf61afa87cf68493995907bf1a908270bae7c11"}
Jul 27 16:28:50.072691 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:50.072630    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp" podStartSLOduration=1.865556533 podStartE2EDuration="2.072612758s" totalImagesPullingTime="207.056225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:28:50.071669583 +0000 UTC m=+707.696850556" watchObservedRunningTime="2026-07-27 16:28:50.072612758 +0000 UTC m=+707.697793710"
Jul 27 16:28:51.373685 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:51.373640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:28:59.069286 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:59.069188    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:59.069286 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:59.069261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:59.082195 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:59.082159    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:28:59.108139 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:28:59.108108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp"
Jul 27 16:30:18.458263 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:30:18.458219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:30:27.874059 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:30:27.873955    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:30:38.468969 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:30:38.468920    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:32:02.793479 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:32:02.793447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:32:02.794495 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:32:02.794474    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:32:19.758921 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:32:19.758878    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:32:28.971691 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:32:28.971654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:32:59.549301 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:32:59.549259    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:33:08.762974 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:33:08.762937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:33:17.671671 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:33:17.671629    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:33:29.667450 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:33:29.667367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:33:40.060018 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:33:40.059972    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:33:50.967843 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:33:50.967806    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:34:01.856741 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:34:01.856701    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:34:11.757108 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:34:11.757063    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:34:34.365579 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:34:34.365542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:34:43.851391 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:34:43.851338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:35:08.565762 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:35:08.565719    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:35:19.154680 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:35:19.154635    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:35:33.867833 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:35:33.867794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:35:45.059166 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:35:45.059124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:35:55.060872 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:35:55.060831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:36:06.969153 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:36:06.969109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:37:02.822500 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:37:02.822469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:37:02.825210 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:37:02.825182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:38:02.855729 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:38:02.855685    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:38:11.156128 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:38:11.156088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:38:24.658772 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:38:24.658738    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:38:38.057324 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:38:38.057288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:38:48.460287 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:38:48.460245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:38:59.260165 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:38:59.260125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:39:07.762445 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:39:07.762404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:39:19.210063 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:39:19.210029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:39:29.969361 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:39:29.969276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:39:39.158024 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:39:39.157968    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:39:50.353902 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:39:50.353866    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:40:11.265022 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:11.264969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:40:20.959175 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:20.959133    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:40:26.561356 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.561311    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"]
Jul 27 16:40:26.565625 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.565599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.568382 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.568353    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-06cd93a0-kserve-self-signed-certs\""
Jul 27 16:40:26.580830 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.580799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"]
Jul 27 16:40:26.706991 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.706955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-dshm\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.706991 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.707041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbbb\" (UniqueName: \"kubernetes.io/projected/0251350e-6439-44ed-a3bb-fc5522c2e900-kube-api-access-8mbbb\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.707277 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.707106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-tmp-dir\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.707277 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.707185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-model-cache\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.707277 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.707219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0251350e-6439-44ed-a3bb-fc5522c2e900-tls-certs\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.707277 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.707241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-home\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.808502 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.808461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-dshm\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.808502 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.808507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbbb\" (UniqueName: \"kubernetes.io/projected/0251350e-6439-44ed-a3bb-fc5522c2e900-kube-api-access-8mbbb\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.808796 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.808564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-tmp-dir\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.808796 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.808618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-model-cache\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.808796 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.808647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0251350e-6439-44ed-a3bb-fc5522c2e900-tls-certs\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.808796 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.808673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-home\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.809126 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.809088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-tmp-dir\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.809126 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.809117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-model-cache\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.809300 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.809152    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-home\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.811312 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.811276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-dshm\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.811565 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.811523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0251350e-6439-44ed-a3bb-fc5522c2e900-tls-certs\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.817814 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.817781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbbb\" (UniqueName: \"kubernetes.io/projected/0251350e-6439-44ed-a3bb-fc5522c2e900-kube-api-access-8mbbb\") pod \"e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") " pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:26.876700 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:26.876647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:27.023793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:27.023698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"]
Jul 27 16:40:27.026736 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:40:27.026699    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0251350e_6439_44ed_a3bb_fc5522c2e900.slice/crio-c6b3f6b7c50330fae01c69df74c8605497d59c91ec5e01653b58c0d5cb831881 WatchSource:0}: Error finding container c6b3f6b7c50330fae01c69df74c8605497d59c91ec5e01653b58c0d5cb831881: Status 404 returned error can't find the container with id c6b3f6b7c50330fae01c69df74c8605497d59c91ec5e01653b58c0d5cb831881
Jul 27 16:40:27.297746 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:27.297708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9" event={"ID":"0251350e-6439-44ed-a3bb-fc5522c2e900","Type":"ContainerStarted","Data":"9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5"}
Jul 27 16:40:27.297746 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:27.297756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9" event={"ID":"0251350e-6439-44ed-a3bb-fc5522c2e900","Type":"ContainerStarted","Data":"c6b3f6b7c50330fae01c69df74c8605497d59c91ec5e01653b58c0d5cb831881"}
Jul 27 16:40:27.315582 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:27.315522    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9" podStartSLOduration=1.315504583 podStartE2EDuration="1.315504583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:40:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:40:27.314832638 +0000 UTC m=+1404.940013603" watchObservedRunningTime="2026-07-27 16:40:27.315504583 +0000 UTC m=+1404.940685536"
Jul 27 16:40:30.599839 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:30.599801    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:40:36.877771 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:36.877663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:36.877771 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:36.877716    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:36.890640 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:36.890609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:40:37.363467 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:40:37.363432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:41:01.664586 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.664551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"]
Jul 27 16:41:01.668080 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.668063    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.670735 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.670719    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-06cd93a0-kserve-self-signed-certs\""
Jul 27 16:41:01.679463 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.679418    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"]
Jul 27 16:41:01.742230 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.742183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tmp-dir\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.742230 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.742233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-dshm\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.742516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.742262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tls-certs\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.742516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.742336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnc7\" (UniqueName: \"kubernetes.io/projected/1dd7e380-76b2-44ff-921c-0dd96635a7e7-kube-api-access-9lnc7\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.742516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.742441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-model-cache\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.742516 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.742477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-home\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.843675 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.843629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnc7\" (UniqueName: \"kubernetes.io/projected/1dd7e380-76b2-44ff-921c-0dd96635a7e7-kube-api-access-9lnc7\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.843896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.843695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-model-cache\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.843896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.843719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-home\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.843896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.843776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tmp-dir\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.843896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.843799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-dshm\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.843896 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.843832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tls-certs\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.844206 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.844176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-model-cache\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.844265 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.844221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-home\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.844327 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.844283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tmp-dir\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.846425 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.846401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-dshm\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.846673 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.846658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tls-certs\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.853891 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.853859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnc7\" (UniqueName: \"kubernetes.io/projected/1dd7e380-76b2-44ff-921c-0dd96635a7e7-kube-api-access-9lnc7\") pod \"e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") " pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:01.980226 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:01.980165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:02.128830 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:02.128792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"]
Jul 27 16:41:02.131971 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:41:02.131939    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dd7e380_76b2_44ff_921c_0dd96635a7e7.slice/crio-be41a0426474b96b002953da599762623547ea9370ed5611ebf02dab8a2fc30f WatchSource:0}: Error finding container be41a0426474b96b002953da599762623547ea9370ed5611ebf02dab8a2fc30f: Status 404 returned error can't find the container with id be41a0426474b96b002953da599762623547ea9370ed5611ebf02dab8a2fc30f
Jul 27 16:41:02.477811 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:02.477769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd" event={"ID":"1dd7e380-76b2-44ff-921c-0dd96635a7e7","Type":"ContainerStarted","Data":"45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb"}
Jul 27 16:41:02.477811 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:02.477816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd" event={"ID":"1dd7e380-76b2-44ff-921c-0dd96635a7e7","Type":"ContainerStarted","Data":"be41a0426474b96b002953da599762623547ea9370ed5611ebf02dab8a2fc30f"}
Jul 27 16:41:02.499146 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:02.499056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd" podStartSLOduration=1.499038882 podStartE2EDuration="1.499038882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:41:02.497559979 +0000 UTC m=+1440.122740932" watchObservedRunningTime="2026-07-27 16:41:02.499038882 +0000 UTC m=+1440.124219836"
Jul 27 16:41:05.751177 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:05.751137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:41:11.980743 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:11.980697    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:11.981219 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:11.980775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:11.993894 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:11.993861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:12.538162 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:12.538125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:41:55.352587 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:55.352537    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"]
Jul 27 16:41:56.874266 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:56.874224    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"]
Jul 27 16:41:58.058401 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:41:58.058358    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:42:02.851476 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:02.851447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:42:02.854758 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:02.854734    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:42:10.372982 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.372916    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9" podUID="0251350e-6439-44ed-a3bb-fc5522c2e900" containerName="main" containerID="cri-o://9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5" gracePeriod=45
Jul 27 16:42:10.633997 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.633914    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:42:10.710720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.710677    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mbbb\" (UniqueName: \"kubernetes.io/projected/0251350e-6439-44ed-a3bb-fc5522c2e900-kube-api-access-8mbbb\") pod \"0251350e-6439-44ed-a3bb-fc5522c2e900\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") "
Jul 27 16:42:10.710720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.710718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-model-cache\") pod \"0251350e-6439-44ed-a3bb-fc5522c2e900\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") "
Jul 27 16:42:10.710950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.710850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-dshm\") pod \"0251350e-6439-44ed-a3bb-fc5522c2e900\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") "
Jul 27 16:42:10.710950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.710909    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0251350e-6439-44ed-a3bb-fc5522c2e900-tls-certs\") pod \"0251350e-6439-44ed-a3bb-fc5522c2e900\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") "
Jul 27 16:42:10.710950 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.710946    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-home\") pod \"0251350e-6439-44ed-a3bb-fc5522c2e900\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") "
Jul 27 16:42:10.711109 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.710986    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-tmp-dir\") pod \"0251350e-6439-44ed-a3bb-fc5522c2e900\" (UID: \"0251350e-6439-44ed-a3bb-fc5522c2e900\") "
Jul 27 16:42:10.711109 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.711064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-model-cache" (OuterVolumeSpecName: "model-cache") pod "0251350e-6439-44ed-a3bb-fc5522c2e900" (UID: "0251350e-6439-44ed-a3bb-fc5522c2e900"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:10.711247 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.711220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-home" (OuterVolumeSpecName: "home") pod "0251350e-6439-44ed-a3bb-fc5522c2e900" (UID: "0251350e-6439-44ed-a3bb-fc5522c2e900"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:10.711347 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.711323    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0251350e-6439-44ed-a3bb-fc5522c2e900" (UID: "0251350e-6439-44ed-a3bb-fc5522c2e900"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:10.711413 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.711397    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:10.711413 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.711411    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:10.713436 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.713400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-dshm" (OuterVolumeSpecName: "dshm") pod "0251350e-6439-44ed-a3bb-fc5522c2e900" (UID: "0251350e-6439-44ed-a3bb-fc5522c2e900"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:10.713436 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.713407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0251350e-6439-44ed-a3bb-fc5522c2e900-kube-api-access-8mbbb" (OuterVolumeSpecName: "kube-api-access-8mbbb") pod "0251350e-6439-44ed-a3bb-fc5522c2e900" (UID: "0251350e-6439-44ed-a3bb-fc5522c2e900"). InnerVolumeSpecName "kube-api-access-8mbbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:42:10.713603 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.713476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0251350e-6439-44ed-a3bb-fc5522c2e900-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0251350e-6439-44ed-a3bb-fc5522c2e900" (UID: "0251350e-6439-44ed-a3bb-fc5522c2e900"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:42:10.812434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.812394    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mbbb\" (UniqueName: \"kubernetes.io/projected/0251350e-6439-44ed-a3bb-fc5522c2e900-kube-api-access-8mbbb\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:10.812434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.812426    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:10.812434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.812438    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0251350e-6439-44ed-a3bb-fc5522c2e900-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:10.812434 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.812447    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0251350e-6439-44ed-a3bb-fc5522c2e900-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:10.818914 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.818880    2579 generic.go:356] "Generic (PLEG): container finished" podID="0251350e-6439-44ed-a3bb-fc5522c2e900" containerID="9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5" exitCode=0
Jul 27 16:42:10.819134 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.818932    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"
Jul 27 16:42:10.819134 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.818952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9" event={"ID":"0251350e-6439-44ed-a3bb-fc5522c2e900","Type":"ContainerDied","Data":"9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5"}
Jul 27 16:42:10.819134 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.818993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9" event={"ID":"0251350e-6439-44ed-a3bb-fc5522c2e900","Type":"ContainerDied","Data":"c6b3f6b7c50330fae01c69df74c8605497d59c91ec5e01653b58c0d5cb831881"}
Jul 27 16:42:10.819134 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.819033    2579 scope.go:117] "RemoveContainer" containerID="9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5"
Jul 27 16:42:10.828940 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.828919    2579 scope.go:117] "RemoveContainer" containerID="9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5"
Jul 27 16:42:10.829307 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:42:10.829288    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5\": container with ID starting with 9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5 not found: ID does not exist" containerID="9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5"
Jul 27 16:42:10.829392 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.829321    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5"} err="failed to get container status \"9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5\": rpc error: code = NotFound desc = could not find container \"9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5\": container with ID starting with 9b4c807a5e56ff7041445aa1fc9a9db8ee751f5c692424dd5ce28cedc68120c5 not found: ID does not exist"
Jul 27 16:42:10.842659 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.842622    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"]
Jul 27 16:42:10.846212 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.846171    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-06cd93a0-kserve-95bfb98b7-brmw9"]
Jul 27 16:42:10.968508 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:10.968471    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0251350e-6439-44ed-a3bb-fc5522c2e900" path="/var/lib/kubelet/pods/0251350e-6439-44ed-a3bb-fc5522c2e900/volumes"
Jul 27 16:42:11.895217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:11.895172    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd" podUID="1dd7e380-76b2-44ff-921c-0dd96635a7e7" containerName="main" containerID="cri-o://45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb" gracePeriod=45
Jul 27 16:42:12.150965 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.150881    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:42:12.225558 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lnc7\" (UniqueName: \"kubernetes.io/projected/1dd7e380-76b2-44ff-921c-0dd96635a7e7-kube-api-access-9lnc7\") pod \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") "
Jul 27 16:42:12.225757 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tls-certs\") pod \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") "
Jul 27 16:42:12.225757 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tmp-dir\") pod \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") "
Jul 27 16:42:12.225757 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225664    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-model-cache\") pod \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") "
Jul 27 16:42:12.225757 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-dshm\") pod \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") "
Jul 27 16:42:12.225948 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-home\") pod \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\" (UID: \"1dd7e380-76b2-44ff-921c-0dd96635a7e7\") "
Jul 27 16:42:12.226036 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1dd7e380-76b2-44ff-921c-0dd96635a7e7" (UID: "1dd7e380-76b2-44ff-921c-0dd96635a7e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:12.226078 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.225982    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-model-cache" (OuterVolumeSpecName: "model-cache") pod "1dd7e380-76b2-44ff-921c-0dd96635a7e7" (UID: "1dd7e380-76b2-44ff-921c-0dd96635a7e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:12.226140 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.226082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-home" (OuterVolumeSpecName: "home") pod "1dd7e380-76b2-44ff-921c-0dd96635a7e7" (UID: "1dd7e380-76b2-44ff-921c-0dd96635a7e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:12.226192 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.226138    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:12.227993 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.227968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-dshm" (OuterVolumeSpecName: "dshm") pod "1dd7e380-76b2-44ff-921c-0dd96635a7e7" (UID: "1dd7e380-76b2-44ff-921c-0dd96635a7e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:42:12.228084 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.228033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dd7e380-76b2-44ff-921c-0dd96635a7e7-kube-api-access-9lnc7" (OuterVolumeSpecName: "kube-api-access-9lnc7") pod "1dd7e380-76b2-44ff-921c-0dd96635a7e7" (UID: "1dd7e380-76b2-44ff-921c-0dd96635a7e7"). InnerVolumeSpecName "kube-api-access-9lnc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:42:12.228121 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.228089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1dd7e380-76b2-44ff-921c-0dd96635a7e7" (UID: "1dd7e380-76b2-44ff-921c-0dd96635a7e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:42:12.327332 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.327288    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:12.327332 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.327320    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:12.327332 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.327331    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dd7e380-76b2-44ff-921c-0dd96635a7e7-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:12.327332 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.327340    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnc7\" (UniqueName: \"kubernetes.io/projected/1dd7e380-76b2-44ff-921c-0dd96635a7e7-kube-api-access-9lnc7\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:12.327626 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.327349    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dd7e380-76b2-44ff-921c-0dd96635a7e7-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:42:12.830972 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.830929    2579 generic.go:356] "Generic (PLEG): container finished" podID="1dd7e380-76b2-44ff-921c-0dd96635a7e7" containerID="45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb" exitCode=0
Jul 27 16:42:12.831217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.830985    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"
Jul 27 16:42:12.831217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.831032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd" event={"ID":"1dd7e380-76b2-44ff-921c-0dd96635a7e7","Type":"ContainerDied","Data":"45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb"}
Jul 27 16:42:12.831217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.831086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd" event={"ID":"1dd7e380-76b2-44ff-921c-0dd96635a7e7","Type":"ContainerDied","Data":"be41a0426474b96b002953da599762623547ea9370ed5611ebf02dab8a2fc30f"}
Jul 27 16:42:12.831217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.831108    2579 scope.go:117] "RemoveContainer" containerID="45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb"
Jul 27 16:42:12.841304 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.841282    2579 scope.go:117] "RemoveContainer" containerID="45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb"
Jul 27 16:42:12.841756 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:42:12.841734    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb\": container with ID starting with 45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb not found: ID does not exist" containerID="45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb"
Jul 27 16:42:12.841858 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.841765    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb"} err="failed to get container status \"45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb\": rpc error: code = NotFound desc = could not find container \"45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb\": container with ID starting with 45e66c7c3743232f9d63b129cca57e4829e3f3e52121461381ba18f1972729eb not found: ID does not exist"
Jul 27 16:42:12.864789 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.864742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"]
Jul 27 16:42:12.875696 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.875664    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-06cd93a0-kserve-7d5f4976cf-xnzhd"]
Jul 27 16:42:12.967783 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:12.967742    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dd7e380-76b2-44ff-921c-0dd96635a7e7" path="/var/lib/kubelet/pods/1dd7e380-76b2-44ff-921c-0dd96635a7e7/volumes"
Jul 27 16:42:13.464441 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:13.464406    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:42:22.558776 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:22.558735    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:42:34.364682 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:34.364642    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:42:45.953320 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:45.953281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:42:55.165893 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:42:55.165847    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:43:05.557371 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:43:05.557327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:43:14.971509 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:43:14.971471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:43:24.859857 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:43:24.859815    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:43:35.970381 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:43:35.970350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:43:52.771375 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:43:52.771329    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:44:01.560849 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:44:01.560802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:44:11.301747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:44:11.301707    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:44:22.884802 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:44:22.884767    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:44:40.864048 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:44:40.863988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:44:53.060705 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:44:53.060665    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:45:05.959761 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:05.959722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:45:15.576935 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:15.576898    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:45:30.073097 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:30.073018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:45:39.674779 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:39.674737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:45:44.602506 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.602461    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"]
Jul 27 16:45:44.603328 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.603300    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0251350e-6439-44ed-a3bb-fc5522c2e900" containerName="main"
Jul 27 16:45:44.603328 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.603329    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0251350e-6439-44ed-a3bb-fc5522c2e900" containerName="main"
Jul 27 16:45:44.603495 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.603349    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dd7e380-76b2-44ff-921c-0dd96635a7e7" containerName="main"
Jul 27 16:45:44.603495 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.603358    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dd7e380-76b2-44ff-921c-0dd96635a7e7" containerName="main"
Jul 27 16:45:44.603577 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.603561    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dd7e380-76b2-44ff-921c-0dd96635a7e7" containerName="main"
Jul 27 16:45:44.603622 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.603585    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0251350e-6439-44ed-a3bb-fc5522c2e900" containerName="main"
Jul 27 16:45:44.608534 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.608499    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.611429 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.611408    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a178d363-openshift-default-dockercfg-6kfdg\""
Jul 27 16:45:44.617925 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.617893    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"]
Jul 27 16:45:44.736965 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.736931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5731195-df34-4b2c-bac6-d96675368934-istiod-ca-cert\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.736965 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.736973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-certs\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.736995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-data\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.737027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr7ck\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-kube-api-access-pr7ck\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.737097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-credential-socket\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.737137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-socket\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.737194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-envoy\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737220 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.737219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-istio-token\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.737401 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.737309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5731195-df34-4b2c-bac6-d96675368934-istio-podinfo\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.838800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.838755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-istio-token\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.838800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.838804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5731195-df34-4b2c-bac6-d96675368934-istio-podinfo\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.838842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5731195-df34-4b2c-bac6-d96675368934-istiod-ca-cert\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.838876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-certs\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.838895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-data\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.838914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr7ck\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-kube-api-access-pr7ck\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-credential-socket\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-socket\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839459 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-envoy\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839459 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-certs\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839557 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-data\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839557 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-credential-socket\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-socket\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.839710 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.839699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5731195-df34-4b2c-bac6-d96675368934-istiod-ca-cert\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.841663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.841630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-envoy\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.841772 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.841704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5731195-df34-4b2c-bac6-d96675368934-istio-podinfo\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.849033 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.848983    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-istio-token\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.849192 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.849134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr7ck\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-kube-api-access-pr7ck\") pod \"e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") " pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:44.921263 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:44.921175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:45.077544 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.077518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"]
Jul 27 16:45:45.080409 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:45:45.080378    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5731195_df34_4b2c_bac6_d96675368934.slice/crio-89c0e82a6f46527098caa08e90f5b435fa139f45e7fdf74370620aeda552d8a6 WatchSource:0}: Error finding container 89c0e82a6f46527098caa08e90f5b435fa139f45e7fdf74370620aeda552d8a6: Status 404 returned error can't find the container with id 89c0e82a6f46527098caa08e90f5b435fa139f45e7fdf74370620aeda552d8a6
Jul 27 16:45:45.082902 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.082865    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:45:45.083038 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.082943    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:45:45.083038 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.082975    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:45:45.839146 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.839104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" event={"ID":"d5731195-df34-4b2c-bac6-d96675368934","Type":"ContainerStarted","Data":"b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"}
Jul 27 16:45:45.839683 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.839173    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" event={"ID":"d5731195-df34-4b2c-bac6-d96675368934","Type":"ContainerStarted","Data":"89c0e82a6f46527098caa08e90f5b435fa139f45e7fdf74370620aeda552d8a6"}
Jul 27 16:45:45.863723 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.863669    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podStartSLOduration=1.8636498540000002 podStartE2EDuration="1.863649854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:45.861808723 +0000 UTC m=+1723.486989677" watchObservedRunningTime="2026-07-27 16:45:45.863649854 +0000 UTC m=+1723.488830807"
Jul 27 16:45:45.921863 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:45.921824    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:46.927514 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:46.927481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:47.848907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:47.848875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:47.849959 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:47.849939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:50.264182 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:50.264141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:45:58.904156 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:58.904059    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5731195-df34-4b2c-bac6-d96675368934" containerID="b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1" exitCode=0
Jul 27 16:45:58.904156 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:58.904134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" event={"ID":"d5731195-df34-4b2c-bac6-d96675368934","Type":"ContainerDied","Data":"b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"}
Jul 27 16:45:58.904644 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:58.904585    2579 scope.go:117] "RemoveContainer" containerID="b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"
Jul 27 16:45:58.905838 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:58.905803    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:45:58.905921 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:58.905904    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:45:58.905966 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:58.905947    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:45:59.915246 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:59.915189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" event={"ID":"d5731195-df34-4b2c-bac6-d96675368934","Type":"ContainerStarted","Data":"e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4"}
Jul 27 16:45:59.921572 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:59.921534    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:45:59.923159 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:59.923119    2579 patch_prober.go:28] interesting pod/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 27 16:45:59.923348 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:45:59.923241    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 27 16:46:00.922193 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:00.922147    2579 patch_prober.go:28] interesting pod/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 27 16:46:00.922720 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:00.922223    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 27 16:46:01.922682 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:01.922625    2579 patch_prober.go:28] interesting pod/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 27 16:46:01.923242 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:01.922709    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 27 16:46:02.849218 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:02.849181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:46:02.922380 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:02.922330    2579 patch_prober.go:28] interesting pod/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 27 16:46:02.923301 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:02.923265    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 27 16:46:03.921870 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:03.921817    2579 patch_prober.go:28] interesting pod/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Jul 27 16:46:03.922125 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:03.921912    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Jul 27 16:46:04.926167 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:04.926128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:46:04.927209 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:04.927185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:46:26.472772 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:26.472729    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:46:27.993575 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:27.993538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"]
Jul 27 16:46:27.994054 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:27.993794    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy" containerID="cri-o://e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4" gracePeriod=30
Jul 27 16:46:33.250953 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.250923    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:46:33.418793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-certs\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.418793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5731195-df34-4b2c-bac6-d96675368934-istiod-ca-cert\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.418793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-envoy\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.418793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-socket\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.419178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418817    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pr7ck\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-kube-api-access-pr7ck\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.419178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5731195-df34-4b2c-bac6-d96675368934-istio-podinfo\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.419178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418939    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-data\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.419178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418960    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-istio-token\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.419178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:46:33.419428 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419203    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:46:33.419428 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5731195-df34-4b2c-bac6-d96675368934-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:46:33.419428 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.418992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-credential-socket\") pod \"d5731195-df34-4b2c-bac6-d96675368934\" (UID: \"d5731195-df34-4b2c-bac6-d96675368934\") "
Jul 27 16:46:33.419558 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:46:33.419945 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419920    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-credential-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.420085 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419949    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.420085 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419965    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5731195-df34-4b2c-bac6-d96675368934-istiod-ca-cert\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.420085 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.419978    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-workload-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.420429 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.420166    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:46:33.421518 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.421483    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:46:33.421632 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.421509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-kube-api-access-pr7ck" (OuterVolumeSpecName: "kube-api-access-pr7ck") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "kube-api-access-pr7ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:46:33.421632 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.421549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5731195-df34-4b2c-bac6-d96675368934-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:46:33.421705 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.421640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5731195-df34-4b2c-bac6-d96675368934" (UID: "d5731195-df34-4b2c-bac6-d96675368934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:46:33.521178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.521140    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-envoy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.521178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.521177    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr7ck\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-kube-api-access-pr7ck\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.521178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.521189    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5731195-df34-4b2c-bac6-d96675368934-istio-podinfo\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.521178 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.521198    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5731195-df34-4b2c-bac6-d96675368934-istio-data\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:33.521480 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:33.521206    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5731195-df34-4b2c-bac6-d96675368934-istio-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:46:34.089245 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.089214    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5731195-df34-4b2c-bac6-d96675368934" containerID="e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4" exitCode=0
Jul 27 16:46:34.089424 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.089269    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"
Jul 27 16:46:34.089424 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.089306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" event={"ID":"d5731195-df34-4b2c-bac6-d96675368934","Type":"ContainerDied","Data":"e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4"}
Jul 27 16:46:34.089424 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.089368    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt" event={"ID":"d5731195-df34-4b2c-bac6-d96675368934","Type":"ContainerDied","Data":"89c0e82a6f46527098caa08e90f5b435fa139f45e7fdf74370620aeda552d8a6"}
Jul 27 16:46:34.089424 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.089399    2579 scope.go:117] "RemoveContainer" containerID="e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4"
Jul 27 16:46:34.100047 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.100026    2579 scope.go:117] "RemoveContainer" containerID="b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"
Jul 27 16:46:34.114358 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.114323    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"]
Jul 27 16:46:34.116296 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.116072    2579 scope.go:117] "RemoveContainer" containerID="e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4"
Jul 27 16:46:34.116476 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:46:34.116458    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4\": container with ID starting with e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4 not found: ID does not exist" containerID="e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4"
Jul 27 16:46:34.116526 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.116486    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4"} err="failed to get container status \"e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4\": rpc error: code = NotFound desc = could not find container \"e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4\": container with ID starting with e2a7c8fc4b63c4e9e698dfbedb2a869f347cb6b7ffd74909f7c8eafdf71775c4 not found: ID does not exist"
Jul 27 16:46:34.116526 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.116504    2579 scope.go:117] "RemoveContainer" containerID="b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"
Jul 27 16:46:34.116785 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:46:34.116759    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1\": container with ID starting with b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1 not found: ID does not exist" containerID="b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"
Jul 27 16:46:34.116851 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.116797    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1"} err="failed to get container status \"b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1\": rpc error: code = NotFound desc = could not find container \"b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1\": container with ID starting with b089ed20d70a531894dfb3bf1a0027d019bb283293c9c1ea5a530fc5ef36e9b1 not found: ID does not exist"
Jul 27 16:46:34.117770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.117750    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a178d363-openshift-default-7df985d5db-8vdqt"]
Jul 27 16:46:34.967721 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:34.967673    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5731195-df34-4b2c-bac6-d96675368934" path="/var/lib/kubelet/pods/d5731195-df34-4b2c-bac6-d96675368934/volumes"
Jul 27 16:46:38.755078 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:46:38.755041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:47:02.879395 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:47:02.879365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:47:02.883571 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:47:02.883544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:47:12.982706 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:47:12.982666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:47:34.481568 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:47:34.481528    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:48:06.556359 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:48:06.556313    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:48:16.671840 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:48:16.671796    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:49:01.052337 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:01.052293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:49:49.681466 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.681428    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-n5bcx"]
Jul 27 16:49:49.681976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.681798    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy"
Jul 27 16:49:49.681976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.681809    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy"
Jul 27 16:49:49.681976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.681885    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy"
Jul 27 16:49:49.684713 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.684683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.687594 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.687568    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Jul 27 16:49:49.760374 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.760332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gbv\" (UniqueName: \"kubernetes.io/projected/11c966de-371b-47b3-8fd5-56979d4a9807-kube-api-access-42gbv\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.760580 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.760427    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c966de-371b-47b3-8fd5-56979d4a9807-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.760580 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.760453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c966de-371b-47b3-8fd5-56979d4a9807-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.760580 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.760532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/11c966de-371b-47b3-8fd5-56979d4a9807-ready\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.861667 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.861629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c966de-371b-47b3-8fd5-56979d4a9807-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.861667 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.861669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c966de-371b-47b3-8fd5-56979d4a9807-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.861902 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.861725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/11c966de-371b-47b3-8fd5-56979d4a9807-ready\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.861902 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.861771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gbv\" (UniqueName: \"kubernetes.io/projected/11c966de-371b-47b3-8fd5-56979d4a9807-kube-api-access-42gbv\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.867868 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.862350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c966de-371b-47b3-8fd5-56979d4a9807-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.867868 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.862446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/11c966de-371b-47b3-8fd5-56979d4a9807-ready\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.867868 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.862488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c966de-371b-47b3-8fd5-56979d4a9807-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.873839 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.873799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gbv\" (UniqueName: \"kubernetes.io/projected/11c966de-371b-47b3-8fd5-56979d4a9807-kube-api-access-42gbv\") pod \"cni-sysctl-allowlist-ds-n5bcx\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") " pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:49.994555 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:49.994505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:50.007180 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:49:50.007143    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c966de_371b_47b3_8fd5_56979d4a9807.slice/crio-8511a7bd9b870ea57f28cf46c0b3fa4a1bf5bcfb7e39e30b5e8f18350347db7d WatchSource:0}: Error finding container 8511a7bd9b870ea57f28cf46c0b3fa4a1bf5bcfb7e39e30b5e8f18350347db7d: Status 404 returned error can't find the container with id 8511a7bd9b870ea57f28cf46c0b3fa4a1bf5bcfb7e39e30b5e8f18350347db7d
Jul 27 16:49:50.047331 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:50.047290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" event={"ID":"11c966de-371b-47b3-8fd5-56979d4a9807","Type":"ContainerStarted","Data":"8511a7bd9b870ea57f28cf46c0b3fa4a1bf5bcfb7e39e30b5e8f18350347db7d"}
Jul 27 16:49:51.053191 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:51.053147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" event={"ID":"11c966de-371b-47b3-8fd5-56979d4a9807","Type":"ContainerStarted","Data":"3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378"}
Jul 27 16:49:51.053658 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:51.053377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:51.064968 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:51.064940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:49:51.076594 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:51.076521    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" podStartSLOduration=2.076503121 podStartE2EDuration="2.076503121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:49:51.075279511 +0000 UTC m=+1968.700460466" watchObservedRunningTime="2026-07-27 16:49:51.076503121 +0000 UTC m=+1968.701684074"
Jul 27 16:49:52.696572 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:52.696523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-n5bcx"]
Jul 27 16:49:54.067873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:49:54.067823    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" gracePeriod=30
Jul 27 16:50:01.055433 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:01.055384    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:01.056450 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:01.056414    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:01.057448 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:01.057414    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:01.057593 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:01.057456    2579 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 27 16:50:11.054981 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:11.054935    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:11.056064 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:11.056026    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:11.056988 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:11.056955    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:11.057113 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:11.057017    2579 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 27 16:50:21.055459 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:21.055405    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:21.056704 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:21.056669    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:21.057652 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:21.057620    2579 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" cmd=["/bin/bash","-c","test -f /ready/ready"]
Jul 27 16:50:21.057729 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:21.057668    2579 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Jul 27 16:50:24.154552 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.154525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-n5bcx_11c966de-371b-47b3-8fd5-56979d4a9807/kube-multus-additional-cni-plugins/0.log"
Jul 27 16:50:24.156775 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.156756    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:50:24.186204 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gbv\" (UniqueName: \"kubernetes.io/projected/11c966de-371b-47b3-8fd5-56979d4a9807-kube-api-access-42gbv\") pod \"11c966de-371b-47b3-8fd5-56979d4a9807\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") "
Jul 27 16:50:24.186398 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186233    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/11c966de-371b-47b3-8fd5-56979d4a9807-ready\") pod \"11c966de-371b-47b3-8fd5-56979d4a9807\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") "
Jul 27 16:50:24.186398 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c966de-371b-47b3-8fd5-56979d4a9807-tuning-conf-dir\") pod \"11c966de-371b-47b3-8fd5-56979d4a9807\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") "
Jul 27 16:50:24.186398 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c966de-371b-47b3-8fd5-56979d4a9807-cni-sysctl-allowlist\") pod \"11c966de-371b-47b3-8fd5-56979d4a9807\" (UID: \"11c966de-371b-47b3-8fd5-56979d4a9807\") "
Jul 27 16:50:24.186522 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/11c966de-371b-47b3-8fd5-56979d4a9807-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "11c966de-371b-47b3-8fd5-56979d4a9807" (UID: "11c966de-371b-47b3-8fd5-56979d4a9807"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Jul 27 16:50:24.186560 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c966de-371b-47b3-8fd5-56979d4a9807-ready" (OuterVolumeSpecName: "ready") pod "11c966de-371b-47b3-8fd5-56979d4a9807" (UID: "11c966de-371b-47b3-8fd5-56979d4a9807"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:50:24.186691 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c966de-371b-47b3-8fd5-56979d4a9807-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "11c966de-371b-47b3-8fd5-56979d4a9807" (UID: "11c966de-371b-47b3-8fd5-56979d4a9807"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:50:24.186763 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186748    2579 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/11c966de-371b-47b3-8fd5-56979d4a9807-ready\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:50:24.186805 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.186767    2579 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c966de-371b-47b3-8fd5-56979d4a9807-tuning-conf-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:50:24.188745 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.188706    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c966de-371b-47b3-8fd5-56979d4a9807-kube-api-access-42gbv" (OuterVolumeSpecName: "kube-api-access-42gbv") pod "11c966de-371b-47b3-8fd5-56979d4a9807" (UID: "11c966de-371b-47b3-8fd5-56979d4a9807"). InnerVolumeSpecName "kube-api-access-42gbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:50:24.224048 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.223995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-n5bcx_11c966de-371b-47b3-8fd5-56979d4a9807/kube-multus-additional-cni-plugins/0.log"
Jul 27 16:50:24.224228 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.224067    2579 generic.go:356] "Generic (PLEG): container finished" podID="11c966de-371b-47b3-8fd5-56979d4a9807" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378" exitCode=137
Jul 27 16:50:24.224228 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.224142    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx"
Jul 27 16:50:24.224228 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.224169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" event={"ID":"11c966de-371b-47b3-8fd5-56979d4a9807","Type":"ContainerDied","Data":"3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378"}
Jul 27 16:50:24.224333 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.224238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-n5bcx" event={"ID":"11c966de-371b-47b3-8fd5-56979d4a9807","Type":"ContainerDied","Data":"8511a7bd9b870ea57f28cf46c0b3fa4a1bf5bcfb7e39e30b5e8f18350347db7d"}
Jul 27 16:50:24.224333 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.224270    2579 scope.go:117] "RemoveContainer" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378"
Jul 27 16:50:24.234575 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.234553    2579 scope.go:117] "RemoveContainer" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378"
Jul 27 16:50:24.234913 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:50:24.234894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378\": container with ID starting with 3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378 not found: ID does not exist" containerID="3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378"
Jul 27 16:50:24.234961 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.234925    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378"} err="failed to get container status \"3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378\": rpc error: code = NotFound desc = could not find container \"3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378\": container with ID starting with 3ad9c6ad506c039734407e23e2f33c852d47f1a987e93f9c0e114791c3d35378 not found: ID does not exist"
Jul 27 16:50:24.250741 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.250683    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-n5bcx"]
Jul 27 16:50:24.252752 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.252718    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-n5bcx"]
Jul 27 16:50:24.287894 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.287857    2579 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/11c966de-371b-47b3-8fd5-56979d4a9807-cni-sysctl-allowlist\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:50:24.287894 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.287890    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gbv\" (UniqueName: \"kubernetes.io/projected/11c966de-371b-47b3-8fd5-56979d4a9807-kube-api-access-42gbv\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:50:24.968147 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:50:24.968113    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" path="/var/lib/kubelet/pods/11c966de-371b-47b3-8fd5-56979d4a9807/volumes"
Jul 27 16:51:29.981686 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:51:29.981597    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:51:40.272977 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:51:40.272939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:51:51.962160 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:51:51.962118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:52:02.907633 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:02.907600    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:52:02.912125 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:02.912096    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:52:13.873610 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:13.873566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:52:50.854831 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:50.854783    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:52:57.366655 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.366561    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"]
Jul 27 16:52:57.367140 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.367110    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy"
Jul 27 16:52:57.367140 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.367129    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy"
Jul 27 16:52:57.367270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.367158    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins"
Jul 27 16:52:57.367270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.367168    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins"
Jul 27 16:52:57.367368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.367270    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5731195-df34-4b2c-bac6-d96675368934" containerName="istio-proxy"
Jul 27 16:52:57.367368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.367291    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c966de-371b-47b3-8fd5-56979d4a9807" containerName="kube-multus-additional-cni-plugins"
Jul 27 16:52:57.370566 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.370541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.373800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.373779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dc13869f\"/\"default-dockercfg-fnhd8\""
Jul 27 16:52:57.374753 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.374735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dc13869f\"/\"openshift-service-ca.crt\""
Jul 27 16:52:57.374827 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.374754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-dc13869f\"/\"e2e-lifecycle-model-dc13869f-kserve-self-signed-certs\""
Jul 27 16:52:57.374827 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.374760    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-dc13869f\"/\"kube-root-ca.crt\""
Jul 27 16:52:57.383321 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.383273    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"]
Jul 27 16:52:57.450211 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.450173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpr4\" (UniqueName: \"kubernetes.io/projected/f98bc1f8-de06-42f4-9088-e658f9253e1e-kube-api-access-8qpr4\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.450211 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.450229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-tmp-dir\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.450465 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.450297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98bc1f8-de06-42f4-9088-e658f9253e1e-tls-certs\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.450465 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.450337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-home\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.450465 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.450424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-dshm\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.450465 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.450452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-model-cache\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.551907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.551862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-dshm\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.551907 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.551911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-model-cache\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.551969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpr4\" (UniqueName: \"kubernetes.io/projected/f98bc1f8-de06-42f4-9088-e658f9253e1e-kube-api-access-8qpr4\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.552026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-tmp-dir\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.552052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98bc1f8-de06-42f4-9088-e658f9253e1e-tls-certs\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552217 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.552086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-home\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552462 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.552434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-model-cache\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552462 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.552450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-tmp-dir\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.552587 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.552478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-home\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.554791 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.554715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-dshm\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.554932 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.554906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98bc1f8-de06-42f4-9088-e658f9253e1e-tls-certs\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.562848 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.562799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpr4\" (UniqueName: \"kubernetes.io/projected/f98bc1f8-de06-42f4-9088-e658f9253e1e-kube-api-access-8qpr4\") pod \"e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") " pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.681416 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.681315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:52:57.829901 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.829870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"]
Jul 27 16:52:57.832972 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:52:57.832934    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf98bc1f8_de06_42f4_9088_e658f9253e1e.slice/crio-470d5311b77b59b14ff8bed2c1df061bb41f0bf73c5007bfaa3094cde22307ca WatchSource:0}: Error finding container 470d5311b77b59b14ff8bed2c1df061bb41f0bf73c5007bfaa3094cde22307ca: Status 404 returned error can't find the container with id 470d5311b77b59b14ff8bed2c1df061bb41f0bf73c5007bfaa3094cde22307ca
Jul 27 16:52:57.965829 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.965792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv" event={"ID":"f98bc1f8-de06-42f4-9088-e658f9253e1e","Type":"ContainerStarted","Data":"3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722"}
Jul 27 16:52:57.966041 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.965838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv" event={"ID":"f98bc1f8-de06-42f4-9088-e658f9253e1e","Type":"ContainerStarted","Data":"470d5311b77b59b14ff8bed2c1df061bb41f0bf73c5007bfaa3094cde22307ca"}
Jul 27 16:52:57.986810 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:52:57.986747    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv" podStartSLOduration=0.986726613 podStartE2EDuration="986.726613ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:52:57.98463635 +0000 UTC m=+2155.609817302" watchObservedRunningTime="2026-07-27 16:52:57.986726613 +0000 UTC m=+2155.611907566"
Jul 27 16:53:02.453156 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:02.453108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:53:07.681641 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:07.681597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:53:07.682197 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:07.681767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:53:07.694362 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:07.694330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:53:08.027179 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:08.027144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:53:24.957020 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:24.956973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:53:43.658649 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:43.658605    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:53:52.547564 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:52.547524    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:53:57.947494 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:53:57.947459    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"]
Jul 27 16:54:12.967812 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:12.967745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv" podUID="f98bc1f8-de06-42f4-9088-e658f9253e1e" containerName="main" containerID="cri-o://3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722" gracePeriod=45
Jul 27 16:54:13.224858 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.224776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:54:13.269283 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98bc1f8-de06-42f4-9088-e658f9253e1e-tls-certs\") pod \"f98bc1f8-de06-42f4-9088-e658f9253e1e\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") "
Jul 27 16:54:13.269489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-home\") pod \"f98bc1f8-de06-42f4-9088-e658f9253e1e\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") "
Jul 27 16:54:13.269489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qpr4\" (UniqueName: \"kubernetes.io/projected/f98bc1f8-de06-42f4-9088-e658f9253e1e-kube-api-access-8qpr4\") pod \"f98bc1f8-de06-42f4-9088-e658f9253e1e\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") "
Jul 27 16:54:13.269489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-tmp-dir\") pod \"f98bc1f8-de06-42f4-9088-e658f9253e1e\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") "
Jul 27 16:54:13.269489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-model-cache\") pod \"f98bc1f8-de06-42f4-9088-e658f9253e1e\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") "
Jul 27 16:54:13.269489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-dshm\") pod \"f98bc1f8-de06-42f4-9088-e658f9253e1e\" (UID: \"f98bc1f8-de06-42f4-9088-e658f9253e1e\") "
Jul 27 16:54:13.269768 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-home" (OuterVolumeSpecName: "home") pod "f98bc1f8-de06-42f4-9088-e658f9253e1e" (UID: "f98bc1f8-de06-42f4-9088-e658f9253e1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:13.269822 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.269783    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:54:13.270158 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.270126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f98bc1f8-de06-42f4-9088-e658f9253e1e" (UID: "f98bc1f8-de06-42f4-9088-e658f9253e1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:13.270547 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.270524    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f98bc1f8-de06-42f4-9088-e658f9253e1e" (UID: "f98bc1f8-de06-42f4-9088-e658f9253e1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:13.272114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.272062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-dshm" (OuterVolumeSpecName: "dshm") pod "f98bc1f8-de06-42f4-9088-e658f9253e1e" (UID: "f98bc1f8-de06-42f4-9088-e658f9253e1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:54:13.272224 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.272159    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f98bc1f8-de06-42f4-9088-e658f9253e1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f98bc1f8-de06-42f4-9088-e658f9253e1e" (UID: "f98bc1f8-de06-42f4-9088-e658f9253e1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:54:13.272224 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.272170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98bc1f8-de06-42f4-9088-e658f9253e1e-kube-api-access-8qpr4" (OuterVolumeSpecName: "kube-api-access-8qpr4") pod "f98bc1f8-de06-42f4-9088-e658f9253e1e" (UID: "f98bc1f8-de06-42f4-9088-e658f9253e1e"). InnerVolumeSpecName "kube-api-access-8qpr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:54:13.327323 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.327287    2579 generic.go:356] "Generic (PLEG): container finished" podID="f98bc1f8-de06-42f4-9088-e658f9253e1e" containerID="3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722" exitCode=0
Jul 27 16:54:13.327513 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.327334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"
Jul 27 16:54:13.327513 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.327354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv" event={"ID":"f98bc1f8-de06-42f4-9088-e658f9253e1e","Type":"ContainerDied","Data":"3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722"}
Jul 27 16:54:13.327513 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.327386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv" event={"ID":"f98bc1f8-de06-42f4-9088-e658f9253e1e","Type":"ContainerDied","Data":"470d5311b77b59b14ff8bed2c1df061bb41f0bf73c5007bfaa3094cde22307ca"}
Jul 27 16:54:13.327513 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.327413    2579 scope.go:117] "RemoveContainer" containerID="3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722"
Jul 27 16:54:13.337167 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.337146    2579 scope.go:117] "RemoveContainer" containerID="3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722"
Jul 27 16:54:13.337552 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:54:13.337531    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722\": container with ID starting with 3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722 not found: ID does not exist" containerID="3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722"
Jul 27 16:54:13.337595 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.337565    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722"} err="failed to get container status \"3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722\": rpc error: code = NotFound desc = could not find container \"3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722\": container with ID starting with 3d54bb2f80dfb268daa33b43cf230eee91aac07fe76847eb1f1b8c91393b3722 not found: ID does not exist"
Jul 27 16:54:13.350132 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.350094    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"]
Jul 27 16:54:13.354401 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.354368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-dc13869f/e2e-lifecycle-model-dc13869f-kserve-66fc7946dc-5m2zv"]
Jul 27 16:54:13.370502 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.370470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpr4\" (UniqueName: \"kubernetes.io/projected/f98bc1f8-de06-42f4-9088-e658f9253e1e-kube-api-access-8qpr4\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:54:13.370620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.370520    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:54:13.370620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.370533    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:54:13.370620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.370542    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f98bc1f8-de06-42f4-9088-e658f9253e1e-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:54:13.370620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:13.370550    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f98bc1f8-de06-42f4-9088-e658f9253e1e-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:54:14.967407 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:14.967369    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f98bc1f8-de06-42f4-9088-e658f9253e1e" path="/var/lib/kubelet/pods/f98bc1f8-de06-42f4-9088-e658f9253e1e/volumes"
Jul 27 16:54:46.884675 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:46.884632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:54:56.871473 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:54:56.871433    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:55:49.473325 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.473274    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"]
Jul 27 16:55:49.473803 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.473721    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f98bc1f8-de06-42f4-9088-e658f9253e1e" containerName="main"
Jul 27 16:55:49.473803 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.473736    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f98bc1f8-de06-42f4-9088-e658f9253e1e" containerName="main"
Jul 27 16:55:49.473803 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.473807    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f98bc1f8-de06-42f4-9088-e658f9253e1e" containerName="main"
Jul 27 16:55:49.476734 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.476712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.480120 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.480100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5de735-openshift-default-dockercfg-s2rd6\""
Jul 27 16:55:49.493334 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.493298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"]
Jul 27 16:55:49.585849 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.585814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-envoy\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.585856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-certs\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.585929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-data\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.585950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-socket\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.585966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6e8369-a904-421c-8e22-fdd4cd924055-istiod-ca-cert\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586107 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.586059    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-token\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586354 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.586107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-credential-socket\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586354 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.586129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-podinfo\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.586354 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.586156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlczn\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-kube-api-access-mlczn\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.669759 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.669717    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:55:49.687590 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-envoy\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687704 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-certs\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687704 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-data\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687704 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-socket\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687704 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6e8369-a904-421c-8e22-fdd4cd924055-istiod-ca-cert\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-token\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-credential-socket\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-podinfo\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.687885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.687790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlczn\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-kube-api-access-mlczn\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.688150 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.688124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-certs\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.688221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.688145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-credential-socket\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.688273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.688215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-data\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.688273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.688245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-socket\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.688776 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.688706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6e8369-a904-421c-8e22-fdd4cd924055-istiod-ca-cert\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.690371 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.690347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-envoy\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.690595 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.690574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-podinfo\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.698154 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.698117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-token\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.698318 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.698226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlczn\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-kube-api-access-mlczn\") pod \"e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") " pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.787506 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.787416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:49.953505 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.953470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"]
Jul 27 16:55:49.955881 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:55:49.955852    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6e8369_a904_421c_8e22_fdd4cd924055.slice/crio-ba77bd947e1ee2502f3f7c22308c3ccf905caaf448f8f70cdfa97cfc2d704d40 WatchSource:0}: Error finding container ba77bd947e1ee2502f3f7c22308c3ccf905caaf448f8f70cdfa97cfc2d704d40: Status 404 returned error can't find the container with id ba77bd947e1ee2502f3f7c22308c3ccf905caaf448f8f70cdfa97cfc2d704d40
Jul 27 16:55:49.958407 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.958367    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:55:49.958483 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.958466    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:55:49.958531 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:49.958513    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:55:50.794841 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:50.794800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" event={"ID":"0f6e8369-a904-421c-8e22-fdd4cd924055","Type":"ContainerStarted","Data":"85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"}
Jul 27 16:55:50.795275 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:50.794862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" event={"ID":"0f6e8369-a904-421c-8e22-fdd4cd924055","Type":"ContainerStarted","Data":"ba77bd947e1ee2502f3f7c22308c3ccf905caaf448f8f70cdfa97cfc2d704d40"}
Jul 27 16:55:50.818482 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:50.818416    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podStartSLOduration=1.818398806 podStartE2EDuration="1.818398806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:55:50.817614399 +0000 UTC m=+2328.442795352" watchObservedRunningTime="2026-07-27 16:55:50.818398806 +0000 UTC m=+2328.443579759"
Jul 27 16:55:51.787657 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:51.787619    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:51.793112 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:51.793078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:51.799859 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:51.799826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:51.800874 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:51.800856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:55:58.758739 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:55:58.758649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:56:02.790263 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.790226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"]
Jul 27 16:56:02.794040 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.793987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:02.810439 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.810403    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23e152\"/\"kube-root-ca.crt\""
Jul 27 16:56:02.811106 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.811080    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23e152\"/\"test-model-23e152-kserve-self-signed-certs\""
Jul 27 16:56:02.811529 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.811510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23e152\"/\"default-dockercfg-5jmlf\""
Jul 27 16:56:02.813194 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.813165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23e152\"/\"openshift-service-ca.crt\""
Jul 27 16:56:02.817711 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.817676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"]
Jul 27 16:56:02.925027 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.924532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-tmp-dir\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:02.925027 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.924617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlbmg\" (UniqueName: \"kubernetes.io/projected/e08f6cf6-0112-4b51-b264-059bb4833917-kube-api-access-wlbmg\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:02.925027 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.924669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08f6cf6-0112-4b51-b264-059bb4833917-tls-certs\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:02.925027 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.924778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-model-cache\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:02.925027 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.924837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-dshm\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:02.925027 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:02.924873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-home\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.026471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlbmg\" (UniqueName: \"kubernetes.io/projected/e08f6cf6-0112-4b51-b264-059bb4833917-kube-api-access-wlbmg\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.026550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08f6cf6-0112-4b51-b264-059bb4833917-tls-certs\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.026673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-model-cache\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.026738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-dshm\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.026777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-home\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.026822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-tmp-dir\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028163 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.027319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-tmp-dir\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.028751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.028418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-model-cache\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.029224 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.029195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-home\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.030715 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.030683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08f6cf6-0112-4b51-b264-059bb4833917-tls-certs\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.030894 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.030870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-dshm\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.045881 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.045799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlbmg\" (UniqueName: \"kubernetes.io/projected/e08f6cf6-0112-4b51-b264-059bb4833917-kube-api-access-wlbmg\") pod \"test-model-23e152-kserve-545c8b9657-4sdkq\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") " pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.115037 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.103679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:03.310626 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.310504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"]
Jul 27 16:56:03.857489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.857454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq" event={"ID":"e08f6cf6-0112-4b51-b264-059bb4833917","Type":"ContainerStarted","Data":"dfc7be0bbbd204b79374adccfe95e31c2d39bef984dce9a00105536c4a1d859c"}
Jul 27 16:56:03.857489 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.857497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq" event={"ID":"e08f6cf6-0112-4b51-b264-059bb4833917","Type":"ContainerStarted","Data":"155195ff339885e27d01cba1914a1c98242a92eee5e0bd736a0243e1713a245f"}
Jul 27 16:56:03.888229 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:03.888171    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq" podStartSLOduration=1.888150235 podStartE2EDuration="1.888150235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:03.884563417 +0000 UTC m=+2341.509744352" watchObservedRunningTime="2026-07-27 16:56:03.888150235 +0000 UTC m=+2341.513331166"
Jul 27 16:56:06.801782 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.801686    2579 patch_prober.go:28] interesting pod/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:56:06.801782 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.801765    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 16:56:06.873412 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.873374    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerID="85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7" exitCode=0
Jul 27 16:56:06.873610 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.873447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" event={"ID":"0f6e8369-a904-421c-8e22-fdd4cd924055","Type":"ContainerDied","Data":"85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"}
Jul 27 16:56:06.873861 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.873848    2579 scope.go:117] "RemoveContainer" containerID="85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"
Jul 27 16:56:06.874997 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.874967    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:56:06.875090 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.875073    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:56:06.875154 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:06.875131    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:56:07.879781 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:07.879750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" event={"ID":"0f6e8369-a904-421c-8e22-fdd4cd924055","Type":"ContainerStarted","Data":"89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0"}
Jul 27 16:56:08.361153 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:08.361113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:56:08.788215 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:08.788175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:56:08.788501 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:08.788477    2579 patch_prober.go:28] interesting pod/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:56:08.788620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:08.788558    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 16:56:09.787966 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:09.787926    2579 patch_prober.go:28] interesting pod/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:56:09.788413 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:09.787986    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 16:56:10.788505 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:10.788459    2579 patch_prober.go:28] interesting pod/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Jul 27 16:56:10.789059 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:10.788551    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 27 16:56:11.793209 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:11.793163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:56:11.793674 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:11.793604    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:56:11.794430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:11.794410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:56:13.107974 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:13.107923    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:13.108455 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:13.108044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:13.121873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:13.121844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:13.921132 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:13.921100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:56:19.509941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:19.509896    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:56:22.488803 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.488759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"]
Jul 27 16:56:22.493470 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.493446    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.496222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.496196    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5de735\"/\"kube-root-ca.crt\""
Jul 27 16:56:22.497024 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.496990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5de735\"/\"default-dockercfg-v2nbj\""
Jul 27 16:56:22.497148 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.497064    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5de735\"/\"openshift-service-ca.crt\""
Jul 27 16:56:22.497148 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.497070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5de735\"/\"test-model-5de735-kserve-self-signed-certs\""
Jul 27 16:56:22.509328 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.509297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"]
Jul 27 16:56:22.527654 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.527616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-model-cache\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.527654 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.527660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-dshm\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.527876 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.527680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-home\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.527876 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.527801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tls-certs\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.527966 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.527878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tmp-dir\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.527966 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.527922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthkj\" (UniqueName: \"kubernetes.io/projected/b93ddc23-5774-4a4d-8086-8d1fad9cb037-kube-api-access-mthkj\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.628663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.628615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tmp-dir\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.628663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.628668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthkj\" (UniqueName: \"kubernetes.io/projected/b93ddc23-5774-4a4d-8086-8d1fad9cb037-kube-api-access-mthkj\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.628921 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.628709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-model-cache\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.628921 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.628734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-dshm\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.628921 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.628751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-home\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.628921 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.628780    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tls-certs\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.629168 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.629144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tmp-dir\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.629240 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.629214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-home\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.629317 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.629283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-model-cache\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.631333 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.631297    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-dshm\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.631622 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.631605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tls-certs\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.641256 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.641223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthkj\" (UniqueName: \"kubernetes.io/projected/b93ddc23-5774-4a4d-8086-8d1fad9cb037-kube-api-access-mthkj\") pod \"test-model-5de735-kserve-bcff98f87-cphjl\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") " pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.802759 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.802658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:22.972915 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:22.972888    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"]
Jul 27 16:56:22.975095 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:56:22.975065    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93ddc23_5774_4a4d_8086_8d1fad9cb037.slice/crio-23e486fc87ae36d6d2c46fe2ae3f6111d0815cdddfdb823e0747b0bf11f599ff WatchSource:0}: Error finding container 23e486fc87ae36d6d2c46fe2ae3f6111d0815cdddfdb823e0747b0bf11f599ff: Status 404 returned error can't find the container with id 23e486fc87ae36d6d2c46fe2ae3f6111d0815cdddfdb823e0747b0bf11f599ff
Jul 27 16:56:23.958430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:23.958392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" event={"ID":"b93ddc23-5774-4a4d-8086-8d1fad9cb037","Type":"ContainerStarted","Data":"2c96888c8d7086c1b2e7672f73a40618ffcb7cb3d216119d2d6903036d0028c0"}
Jul 27 16:56:23.958430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:23.958435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" event={"ID":"b93ddc23-5774-4a4d-8086-8d1fad9cb037","Type":"ContainerStarted","Data":"23e486fc87ae36d6d2c46fe2ae3f6111d0815cdddfdb823e0747b0bf11f599ff"}
Jul 27 16:56:23.980767 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:23.980709    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" podStartSLOduration=1.98068804 podStartE2EDuration="1.98068804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:56:23.979643801 +0000 UTC m=+2361.604824755" watchObservedRunningTime="2026-07-27 16:56:23.98068804 +0000 UTC m=+2361.605868996"
Jul 27 16:56:32.803773 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:32.803722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:32.803773 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:32.803784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:32.816724 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:32.816693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:33.013116 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:33.013084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:56:36.161559 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:56:36.161514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:57:02.944849 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:02.944811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:57:02.948798 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:02.948764    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 16:57:07.094305 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:07.094271    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"]
Jul 27 16:57:07.988762 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:07.988714    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"]
Jul 27 16:57:11.963081 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:11.963042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:57:22.114518 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.114448    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq" podUID="e08f6cf6-0112-4b51-b264-059bb4833917" containerName="main" containerID="cri-o://dfc7be0bbbd204b79374adccfe95e31c2d39bef984dce9a00105536c4a1d859c" gracePeriod=45
Jul 27 16:57:22.244860 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.244809    2579 generic.go:356] "Generic (PLEG): container finished" podID="e08f6cf6-0112-4b51-b264-059bb4833917" containerID="dfc7be0bbbd204b79374adccfe95e31c2d39bef984dce9a00105536c4a1d859c" exitCode=0
Jul 27 16:57:22.245061 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.244883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq" event={"ID":"e08f6cf6-0112-4b51-b264-059bb4833917","Type":"ContainerDied","Data":"dfc7be0bbbd204b79374adccfe95e31c2d39bef984dce9a00105536c4a1d859c"}
Jul 27 16:57:22.372308 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.372232    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:57:22.424057 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.423993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-dshm\") pod \"e08f6cf6-0112-4b51-b264-059bb4833917\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") "
Jul 27 16:57:22.424270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424094    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-model-cache\") pod \"e08f6cf6-0112-4b51-b264-059bb4833917\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") "
Jul 27 16:57:22.424270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-tmp-dir\") pod \"e08f6cf6-0112-4b51-b264-059bb4833917\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") "
Jul 27 16:57:22.424270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424173    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlbmg\" (UniqueName: \"kubernetes.io/projected/e08f6cf6-0112-4b51-b264-059bb4833917-kube-api-access-wlbmg\") pod \"e08f6cf6-0112-4b51-b264-059bb4833917\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") "
Jul 27 16:57:22.424270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08f6cf6-0112-4b51-b264-059bb4833917-tls-certs\") pod \"e08f6cf6-0112-4b51-b264-059bb4833917\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") "
Jul 27 16:57:22.424270 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-home\") pod \"e08f6cf6-0112-4b51-b264-059bb4833917\" (UID: \"e08f6cf6-0112-4b51-b264-059bb4833917\") "
Jul 27 16:57:22.424525 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-model-cache" (OuterVolumeSpecName: "model-cache") pod "e08f6cf6-0112-4b51-b264-059bb4833917" (UID: "e08f6cf6-0112-4b51-b264-059bb4833917"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:22.424525 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e08f6cf6-0112-4b51-b264-059bb4833917" (UID: "e08f6cf6-0112-4b51-b264-059bb4833917"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:22.424525 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424502    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:22.424664 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.424641    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-home" (OuterVolumeSpecName: "home") pod "e08f6cf6-0112-4b51-b264-059bb4833917" (UID: "e08f6cf6-0112-4b51-b264-059bb4833917"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:22.426641 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.426612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-dshm" (OuterVolumeSpecName: "dshm") pod "e08f6cf6-0112-4b51-b264-059bb4833917" (UID: "e08f6cf6-0112-4b51-b264-059bb4833917"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:22.426641 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.426614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08f6cf6-0112-4b51-b264-059bb4833917-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e08f6cf6-0112-4b51-b264-059bb4833917" (UID: "e08f6cf6-0112-4b51-b264-059bb4833917"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:57:22.426808 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.426674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08f6cf6-0112-4b51-b264-059bb4833917-kube-api-access-wlbmg" (OuterVolumeSpecName: "kube-api-access-wlbmg") pod "e08f6cf6-0112-4b51-b264-059bb4833917" (UID: "e08f6cf6-0112-4b51-b264-059bb4833917"). InnerVolumeSpecName "kube-api-access-wlbmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:22.525873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.525818    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlbmg\" (UniqueName: \"kubernetes.io/projected/e08f6cf6-0112-4b51-b264-059bb4833917-kube-api-access-wlbmg\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:22.525873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.525865    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e08f6cf6-0112-4b51-b264-059bb4833917-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:22.525873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.525875    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:22.525873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.525882    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:22.525873 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:22.525890    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e08f6cf6-0112-4b51-b264-059bb4833917-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:23.009588 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.009543    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" podUID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" containerName="main" containerID="cri-o://2c96888c8d7086c1b2e7672f73a40618ffcb7cb3d216119d2d6903036d0028c0" gracePeriod=45
Jul 27 16:57:23.013869 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.013819    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" podUID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/ready\": EOF"
Jul 27 16:57:23.252952 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.252862    2579 generic.go:356] "Generic (PLEG): container finished" podID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" containerID="2c96888c8d7086c1b2e7672f73a40618ffcb7cb3d216119d2d6903036d0028c0" exitCode=0
Jul 27 16:57:23.252952 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.252898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" event={"ID":"b93ddc23-5774-4a4d-8086-8d1fad9cb037","Type":"ContainerDied","Data":"2c96888c8d7086c1b2e7672f73a40618ffcb7cb3d216119d2d6903036d0028c0"}
Jul 27 16:57:23.254612 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.254578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq" event={"ID":"e08f6cf6-0112-4b51-b264-059bb4833917","Type":"ContainerDied","Data":"155195ff339885e27d01cba1914a1c98242a92eee5e0bd736a0243e1713a245f"}
Jul 27 16:57:23.254612 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.254604    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"
Jul 27 16:57:23.254792 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.254638    2579 scope.go:117] "RemoveContainer" containerID="dfc7be0bbbd204b79374adccfe95e31c2d39bef984dce9a00105536c4a1d859c"
Jul 27 16:57:23.272899 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.272863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"]
Jul 27 16:57:23.275928 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.275895    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-23e152/test-model-23e152-kserve-545c8b9657-4sdkq"]
Jul 27 16:57:23.284667 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.284641    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:57:23.332239 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tls-certs\") pod \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") "
Jul 27 16:57:23.332449 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332261    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-model-cache\") pod \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") "
Jul 27 16:57:23.332449 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-dshm\") pod \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") "
Jul 27 16:57:23.332449 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tmp-dir\") pod \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") "
Jul 27 16:57:23.332449 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-home\") pod \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") "
Jul 27 16:57:23.332449 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mthkj\" (UniqueName: \"kubernetes.io/projected/b93ddc23-5774-4a4d-8086-8d1fad9cb037-kube-api-access-mthkj\") pod \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\" (UID: \"b93ddc23-5774-4a4d-8086-8d1fad9cb037\") "
Jul 27 16:57:23.332743 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332561    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-model-cache" (OuterVolumeSpecName: "model-cache") pod "b93ddc23-5774-4a4d-8086-8d1fad9cb037" (UID: "b93ddc23-5774-4a4d-8086-8d1fad9cb037"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:23.332743 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b93ddc23-5774-4a4d-8086-8d1fad9cb037" (UID: "b93ddc23-5774-4a4d-8086-8d1fad9cb037"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:23.332743 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-home" (OuterVolumeSpecName: "home") pod "b93ddc23-5774-4a4d-8086-8d1fad9cb037" (UID: "b93ddc23-5774-4a4d-8086-8d1fad9cb037"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:23.332884 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332786    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:23.332884 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332809    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:23.332884 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.332827    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:23.334864 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.334827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-dshm" (OuterVolumeSpecName: "dshm") pod "b93ddc23-5774-4a4d-8086-8d1fad9cb037" (UID: "b93ddc23-5774-4a4d-8086-8d1fad9cb037"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:23.334864 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.334837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93ddc23-5774-4a4d-8086-8d1fad9cb037-kube-api-access-mthkj" (OuterVolumeSpecName: "kube-api-access-mthkj") pod "b93ddc23-5774-4a4d-8086-8d1fad9cb037" (UID: "b93ddc23-5774-4a4d-8086-8d1fad9cb037"). InnerVolumeSpecName "kube-api-access-mthkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:23.334864 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.334865    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b93ddc23-5774-4a4d-8086-8d1fad9cb037" (UID: "b93ddc23-5774-4a4d-8086-8d1fad9cb037"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:57:23.433867 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.433824    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93ddc23-5774-4a4d-8086-8d1fad9cb037-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:23.433867 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.433858    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthkj\" (UniqueName: \"kubernetes.io/projected/b93ddc23-5774-4a4d-8086-8d1fad9cb037-kube-api-access-mthkj\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:23.433867 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:23.433874    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93ddc23-5774-4a4d-8086-8d1fad9cb037-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:24.262116 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.262080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl" event={"ID":"b93ddc23-5774-4a4d-8086-8d1fad9cb037","Type":"ContainerDied","Data":"23e486fc87ae36d6d2c46fe2ae3f6111d0815cdddfdb823e0747b0bf11f599ff"}
Jul 27 16:57:24.262116 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.262125    2579 scope.go:117] "RemoveContainer" containerID="2c96888c8d7086c1b2e7672f73a40618ffcb7cb3d216119d2d6903036d0028c0"
Jul 27 16:57:24.262669 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.262088    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"
Jul 27 16:57:24.287808 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.287770    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"]
Jul 27 16:57:24.291129 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.291098    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5de735/test-model-5de735-kserve-bcff98f87-cphjl"]
Jul 27 16:57:24.655137 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.655040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:57:24.968095 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.968060    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" path="/var/lib/kubelet/pods/b93ddc23-5774-4a4d-8086-8d1fad9cb037/volumes"
Jul 27 16:57:24.968571 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:24.968546    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e08f6cf6-0112-4b51-b264-059bb4833917" path="/var/lib/kubelet/pods/e08f6cf6-0112-4b51-b264-059bb4833917/volumes"
Jul 27 16:57:43.859924 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:43.859886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:57:52.605957 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:52.605914    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"]
Jul 27 16:57:52.606497 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:52.606218    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy" containerID="cri-o://89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0" gracePeriod=30
Jul 27 16:57:56.796645 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:56.796604    2579 patch_prober.go:28] interesting pod/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 16:57:56.797103 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:56.796668    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 16:57:57.873405 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.873332    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:57:57.879949 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.879923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6e8369-a904-421c-8e22-fdd4cd924055-istiod-ca-cert\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880148 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.879993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-envoy\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880148 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-certs\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880148 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880069    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-token\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880148 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-data\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880363 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlczn\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-kube-api-access-mlczn\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880363 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-credential-socket\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880460 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:57.880460 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:57.880587 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6e8369-a904-421c-8e22-fdd4cd924055-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:57:57.880587 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880493    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-podinfo\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880587 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-socket\") pod \"0f6e8369-a904-421c-8e22-fdd4cd924055\" (UID: \"0f6e8369-a904-421c-8e22-fdd4cd924055\") "
Jul 27 16:57:57.880773 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:57.881026 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880958    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-credential-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.881026 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880976    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.881026 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880992    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f6e8369-a904-421c-8e22-fdd4cd924055-istiod-ca-cert\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.881185 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.881036    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-workload-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.881185 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.880998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:57.882631 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.882597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:57:57.882631 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.882603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:57.882780 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.882686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-kube-api-access-mlczn" (OuterVolumeSpecName: "kube-api-access-mlczn") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "kube-api-access-mlczn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:57:57.883276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.883258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f6e8369-a904-421c-8e22-fdd4cd924055" (UID: "0f6e8369-a904-421c-8e22-fdd4cd924055"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:57:57.982458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.982410    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-envoy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.982458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.982447    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.982458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.982455    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-data\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.982458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.982464    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlczn\" (UniqueName: \"kubernetes.io/projected/0f6e8369-a904-421c-8e22-fdd4cd924055-kube-api-access-mlczn\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:57.982458 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:57.982475    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f6e8369-a904-421c-8e22-fdd4cd924055-istio-podinfo\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:57:58.425432 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.425389    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerID="89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0" exitCode=0
Jul 27 16:57:58.425620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.425445    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"
Jul 27 16:57:58.425620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.425475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" event={"ID":"0f6e8369-a904-421c-8e22-fdd4cd924055","Type":"ContainerDied","Data":"89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0"}
Jul 27 16:57:58.425620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.425532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct" event={"ID":"0f6e8369-a904-421c-8e22-fdd4cd924055","Type":"ContainerDied","Data":"ba77bd947e1ee2502f3f7c22308c3ccf905caaf448f8f70cdfa97cfc2d704d40"}
Jul 27 16:57:58.425620 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.425558    2579 scope.go:117] "RemoveContainer" containerID="89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0"
Jul 27 16:57:58.435463 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.435445    2579 scope.go:117] "RemoveContainer" containerID="85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"
Jul 27 16:57:58.444695 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.444676    2579 scope.go:117] "RemoveContainer" containerID="89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0"
Jul 27 16:57:58.445027 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:57:58.444988    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0\": container with ID starting with 89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0 not found: ID does not exist" containerID="89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0"
Jul 27 16:57:58.445066 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.445040    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0"} err="failed to get container status \"89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0\": rpc error: code = NotFound desc = could not find container \"89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0\": container with ID starting with 89453fa7d2fee01fd000332bfa0c7c1bddd475dd304ea53e8f56ff6757673be0 not found: ID does not exist"
Jul 27 16:57:58.445066 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.445062    2579 scope.go:117] "RemoveContainer" containerID="85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"
Jul 27 16:57:58.445353 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:57:58.445334    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7\": container with ID starting with 85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7 not found: ID does not exist" containerID="85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"
Jul 27 16:57:58.445393 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.445362    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7"} err="failed to get container status \"85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7\": rpc error: code = NotFound desc = could not find container \"85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7\": container with ID starting with 85ec8d3e280cb92b6a103ea612890df9a71e09c174fe530218dca1d5459882a7 not found: ID does not exist"
Jul 27 16:57:58.474967 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.474931    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"]
Jul 27 16:57:58.496044 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.495982    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5de735-openshift-default-77cf6789b6-k2vct"]
Jul 27 16:57:58.973826 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:57:58.973790    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" path="/var/lib/kubelet/pods/0f6e8369-a904-421c-8e22-fdd4cd924055/volumes"
Jul 27 16:58:02.267716 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.267675    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"]
Jul 27 16:58:02.268161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268095    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" containerName="main"
Jul 27 16:58:02.268161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268107    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" containerName="main"
Jul 27 16:58:02.268161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268119    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy"
Jul 27 16:58:02.268161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268124    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy"
Jul 27 16:58:02.268161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268136    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08f6cf6-0112-4b51-b264-059bb4833917" containerName="main"
Jul 27 16:58:02.268161 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268141    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08f6cf6-0112-4b51-b264-059bb4833917" containerName="main"
Jul 27 16:58:02.268355 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268217    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08f6cf6-0112-4b51-b264-059bb4833917" containerName="main"
Jul 27 16:58:02.268355 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268225    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93ddc23-5774-4a4d-8086-8d1fad9cb037" containerName="main"
Jul 27 16:58:02.268355 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.268230    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy"
Jul 27 16:58:02.272771 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.272748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.276447 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.276424    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-30f382-openshift-default-dockercfg-d9j9g\""
Jul 27 16:58:02.290172 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.290134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"]
Jul 27 16:58:02.323286 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-envoy\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323490 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-socket\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323490 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-credential-socket\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323490 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e64c351-af39-46a9-b6ef-7269fe5c0576-istiod-ca-cert\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323490 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-token\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323490 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-data\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323671 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q674k\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-kube-api-access-q674k\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323671 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-certs\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.323671 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.323573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-podinfo\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424347 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-certs\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424347 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-podinfo\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-envoy\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-socket\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-credential-socket\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e64c351-af39-46a9-b6ef-7269fe5c0576-istiod-ca-cert\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424629 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-token\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424898 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-data\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424898 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q674k\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-kube-api-access-q674k\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424898 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-certs\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424898 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-socket\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.424898 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.424869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-credential-socket\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.425187 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.425162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-data\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.425362 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.425342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e64c351-af39-46a9-b6ef-7269fe5c0576-istiod-ca-cert\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.427182 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.427160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-envoy\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.427322 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.427304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-podinfo\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.444749 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.444713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-token\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.445106 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.445074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q674k\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-kube-api-access-q674k\") pod \"e2e-api-b-30f382-openshift-default-ddff65754-t8dvq\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") " pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.583942 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.583838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:02.768201 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.768162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"]
Jul 27 16:58:02.774978 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:58:02.774948    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e64c351_af39_46a9_b6ef_7269fe5c0576.slice/crio-dc6fa3829978d38b2757b98a8f62bb31d871339670c1dfced075c88c09038b0c WatchSource:0}: Error finding container dc6fa3829978d38b2757b98a8f62bb31d871339670c1dfced075c88c09038b0c: Status 404 returned error can't find the container with id dc6fa3829978d38b2757b98a8f62bb31d871339670c1dfced075c88c09038b0c
Jul 27 16:58:02.777249 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.777198    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:58:02.777354 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.777290    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:58:02.777354 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:02.777322    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:58:03.453265 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:03.453218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" event={"ID":"2e64c351-af39-46a9-b6ef-7269fe5c0576","Type":"ContainerStarted","Data":"63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"}
Jul 27 16:58:03.453679 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:03.453281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" event={"ID":"2e64c351-af39-46a9-b6ef-7269fe5c0576","Type":"ContainerStarted","Data":"dc6fa3829978d38b2757b98a8f62bb31d871339670c1dfced075c88c09038b0c"}
Jul 27 16:58:03.537104 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:03.537053    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podStartSLOduration=1.537033696 podStartE2EDuration="1.537033696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:58:03.5345113 +0000 UTC m=+2461.159692254" watchObservedRunningTime="2026-07-27 16:58:03.537033696 +0000 UTC m=+2461.162214641"
Jul 27 16:58:03.584092 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:03.584047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:03.589500 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:03.589473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:04.357346 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:04.357300    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:58:04.458505 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:04.458472    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:04.459746 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:04.459721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:20.501221 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:20.501178    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 16:58:20.501683 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:20.501257    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 16:58:20.959099 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:20.959058    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:58:22.560853 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:22.560818    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerID="63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad" exitCode=0
Jul 27 16:58:22.561394 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:22.560894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" event={"ID":"2e64c351-af39-46a9-b6ef-7269fe5c0576","Type":"ContainerDied","Data":"63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"}
Jul 27 16:58:22.561468 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:22.561403    2579 scope.go:117] "RemoveContainer" containerID="63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"
Jul 27 16:58:22.562589 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:22.562552    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:58:22.562702 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:22.562649    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:58:22.562702 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:22.562695    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:58:23.567560 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:23.567522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" event={"ID":"2e64c351-af39-46a9-b6ef-7269fe5c0576","Type":"ContainerStarted","Data":"eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3"}
Jul 27 16:58:23.584602 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:23.584567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:23.584905 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:23.584881    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:58:23.584957 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:23.584934    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 27 16:58:24.584608 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:24.584554    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:58:24.585065 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:24.584632    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 27 16:58:25.584325 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:25.584283    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:58:25.584523 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:25.584344    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 27 16:58:26.585222 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:26.585184    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:58:26.585654 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:26.585244    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 27 16:58:27.584274 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:27.584236    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Jul 27 16:58:27.584462 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:27.584306    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Jul 27 16:58:28.588885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:28.588853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:28.589420 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:28.589212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:28.590023 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:28.589981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:58:37.064340 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:58:37.064299    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:59:06.098528 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:06.098477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:59:12.378575 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:12.378534    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"]
Jul 27 16:59:12.378992 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:12.378780    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" containerID="cri-o://eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3" gracePeriod=30
Jul 27 16:59:13.591798 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:13.591760    2579 patch_prober.go:28] interesting pod/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 16:59:13.592192 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:13.591820    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 16:59:17.647827 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.647753    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:59:17.763747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e64c351-af39-46a9-b6ef-7269fe5c0576-istiod-ca-cert\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.763747 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-credential-socket\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-data\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-token\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-socket\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763873    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-certs\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-podinfo\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.763968    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-envoy\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764046 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.764037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q674k\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-kube-api-access-q674k\") pod \"2e64c351-af39-46a9-b6ef-7269fe5c0576\" (UID: \"2e64c351-af39-46a9-b6ef-7269fe5c0576\") "
Jul 27 16:59:17.764368 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.764244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e64c351-af39-46a9-b6ef-7269fe5c0576-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:59:17.764430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.764374    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e64c351-af39-46a9-b6ef-7269fe5c0576-istiod-ca-cert\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.764430 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.764408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:17.764663 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.764611    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:17.764770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.764660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:17.765552 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.765526    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:17.766815 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.766787    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:59:17.766815 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.766794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 16:59:17.766954 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.766843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-kube-api-access-q674k" (OuterVolumeSpecName: "kube-api-access-q674k") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "kube-api-access-q674k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:59:17.767044 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.767025    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e64c351-af39-46a9-b6ef-7269fe5c0576" (UID: "2e64c351-af39-46a9-b6ef-7269fe5c0576"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:59:17.831751 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.831713    2579 generic.go:356] "Generic (PLEG): container finished" podID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerID="eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3" exitCode=0
Jul 27 16:59:17.831976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.831767    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"
Jul 27 16:59:17.831976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.831803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" event={"ID":"2e64c351-af39-46a9-b6ef-7269fe5c0576","Type":"ContainerDied","Data":"eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3"}
Jul 27 16:59:17.831976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.831874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq" event={"ID":"2e64c351-af39-46a9-b6ef-7269fe5c0576","Type":"ContainerDied","Data":"dc6fa3829978d38b2757b98a8f62bb31d871339670c1dfced075c88c09038b0c"}
Jul 27 16:59:17.831976 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.831907    2579 scope.go:117] "RemoveContainer" containerID="eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3"
Jul 27 16:59:17.842067 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.842044    2579 scope.go:117] "RemoveContainer" containerID="63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"
Jul 27 16:59:17.851633 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.851604    2579 scope.go:117] "RemoveContainer" containerID="eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3"
Jul 27 16:59:17.851959 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:59:17.851939    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3\": container with ID starting with eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3 not found: ID does not exist" containerID="eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3"
Jul 27 16:59:17.852057 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.851971    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3"} err="failed to get container status \"eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3\": rpc error: code = NotFound desc = could not find container \"eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3\": container with ID starting with eba5ed7949aeff8cd3e89466f3e0ba0e837fad5c6607645571a9473ee7ff14c3 not found: ID does not exist"
Jul 27 16:59:17.852057 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.851992    2579 scope.go:117] "RemoveContainer" containerID="63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"
Jul 27 16:59:17.852482 ip-10-0-129-154 kubenswrapper[2579]: E0727 16:59:17.852354    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad\": container with ID starting with 63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad not found: ID does not exist" containerID="63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"
Jul 27 16:59:17.852482 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.852387    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad"} err="failed to get container status \"63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad\": rpc error: code = NotFound desc = could not find container \"63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad\": container with ID starting with 63e6e920dd7528dc59ad436af116bdd225f54be85feb06ffe9fc1d508bd7d6ad not found: ID does not exist"
Jul 27 16:59:17.855091 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.855030    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"]
Jul 27 16:59:17.859520 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.859488    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-30f382-openshift-default-ddff65754-t8dvq"]
Jul 27 16:59:17.865798 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865770    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-data\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865798 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865799    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865808    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865817    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-workload-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865851    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-podinfo\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865859    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-istio-envoy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865868    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q674k\" (UniqueName: \"kubernetes.io/projected/2e64c351-af39-46a9-b6ef-7269fe5c0576-kube-api-access-q674k\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:17.865941 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:17.865879    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e64c351-af39-46a9-b6ef-7269fe5c0576-credential-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 16:59:18.966943 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:18.966897    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" path="/var/lib/kubelet/pods/2e64c351-af39-46a9-b6ef-7269fe5c0576/volumes"
Jul 27 16:59:21.473341 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473295    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"]
Jul 27 16:59:21.473793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473696    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy"
Jul 27 16:59:21.473793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473715    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy"
Jul 27 16:59:21.473793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473730    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy"
Jul 27 16:59:21.473793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473739    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy"
Jul 27 16:59:21.473793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473750    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy"
Jul 27 16:59:21.473793 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473759    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy"
Jul 27 16:59:21.474076 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy"
Jul 27 16:59:21.474076 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473858    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e64c351-af39-46a9-b6ef-7269fe5c0576" containerName="istio-proxy"
Jul 27 16:59:21.474076 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.473866    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6e8369-a904-421c-8e22-fdd4cd924055" containerName="istio-proxy"
Jul 27 16:59:21.478828 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.478795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.481689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.481667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d4529a-openshift-default-dockercfg-6kzdv\""
Jul 27 16:59:21.493559 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.493521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"]
Jul 27 16:59:21.601764 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.601712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-socket\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.601764 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.601772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009d373e-8fef-4f9a-84f3-f9c7a7180607-istiod-ca-cert\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.601816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-podinfo\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.601928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-certs\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.601960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-credential-socket\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.601979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-envoy\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.602034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-token\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.602076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfpm\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-kube-api-access-nzfpm\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.602291 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.602119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-data\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.702942 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.702894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-certs\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.702942 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.702941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-credential-socket\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.702967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-envoy\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.702994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-token\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfpm\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-kube-api-access-nzfpm\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-data\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-socket\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009d373e-8fef-4f9a-84f3-f9c7a7180607-istiod-ca-cert\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703273 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-podinfo\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703644 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-certs\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703644 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703619    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-socket\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703759 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-credential-socket\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.703759 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.703743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-data\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.704084 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.704069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009d373e-8fef-4f9a-84f3-f9c7a7180607-istiod-ca-cert\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.705878 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.705853    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-envoy\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.706018 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.705965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-podinfo\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.719380 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.719340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-token\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.719553 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.719482    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfpm\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-kube-api-access-nzfpm\") pod \"e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") " pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.790452 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.790350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:21.951093 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.951060    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"]
Jul 27 16:59:21.954521 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:59:21.954481    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009d373e_8fef_4f9a_84f3_f9c7a7180607.slice/crio-592c03a25cb00178c5dc3896bb2d7985984888ec75fa9e5f1db4e283d88f2538 WatchSource:0}: Error finding container 592c03a25cb00178c5dc3896bb2d7985984888ec75fa9e5f1db4e283d88f2538: Status 404 returned error can't find the container with id 592c03a25cb00178c5dc3896bb2d7985984888ec75fa9e5f1db4e283d88f2538
Jul 27 16:59:21.956712 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.956681    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:59:21.956770 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.956753    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:59:21.956812 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:21.956784    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:59:22.862586 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:22.862544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" event={"ID":"009d373e-8fef-4f9a-84f3-f9c7a7180607","Type":"ContainerStarted","Data":"0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"}
Jul 27 16:59:22.863053 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:22.862632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" event={"ID":"009d373e-8fef-4f9a-84f3-f9c7a7180607","Type":"ContainerStarted","Data":"592c03a25cb00178c5dc3896bb2d7985984888ec75fa9e5f1db4e283d88f2538"}
Jul 27 16:59:22.888519 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:22.888456    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podStartSLOduration=1.8884407269999999 podStartE2EDuration="1.888440727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:22.8875247 +0000 UTC m=+2540.512705655" watchObservedRunningTime="2026-07-27 16:59:22.888440727 +0000 UTC m=+2540.513621680"
Jul 27 16:59:23.791113 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:23.791068    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:23.796694 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:23.796648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:23.867406 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:23.867375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:23.868631 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:23.868612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:24.160494 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:24.160405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:59:36.163695 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:36.163653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:59:39.712285 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.711334    2579 patch_prober.go:28] interesting pod/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:47402->10.134.0.37:15021: read: connection reset by peer" start-of-body=
Jul 27 16:59:39.712285 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.711409    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:47402->10.134.0.37:15021: read: connection reset by peer"
Jul 27 16:59:39.952276 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.952239    2579 generic.go:356] "Generic (PLEG): container finished" podID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerID="0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae" exitCode=0
Jul 27 16:59:39.952451 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.952313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" event={"ID":"009d373e-8fef-4f9a-84f3-f9c7a7180607","Type":"ContainerDied","Data":"0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"}
Jul 27 16:59:39.952838 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.952824    2579 scope.go:117] "RemoveContainer" containerID="0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"
Jul 27 16:59:39.953954 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.953920    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:59:39.954086 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.954051    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:59:39.954153 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:39.954104    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 16:59:40.959557 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:40.959518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" event={"ID":"009d373e-8fef-4f9a-84f3-f9c7a7180607","Type":"ContainerStarted","Data":"94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088"}
Jul 27 16:59:41.790874 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:41.790826    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:41.791142 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:41.791119    2579 patch_prober.go:28] interesting pod/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 27 16:59:41.791197 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:41.791172    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 27 16:59:42.756909 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.756867    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"]
Jul 27 16:59:42.761879 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.761850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:42.765607 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.765580    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f1b699\"/\"auth-test-model-f1b699-kserve-self-signed-certs\""
Jul 27 16:59:42.765785 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.765580    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f1b699\"/\"openshift-service-ca.crt\""
Jul 27 16:59:42.765785 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.765580    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f1b699\"/\"kube-root-ca.crt\""
Jul 27 16:59:42.765926 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.765823    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f1b699\"/\"default-dockercfg-zcwth\""
Jul 27 16:59:42.771058 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.771025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"]
Jul 27 16:59:42.790985 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.790945    2579 patch_prober.go:28] interesting pod/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 27 16:59:42.791200 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.791033    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 27 16:59:42.927689 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.927637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqx8l\" (UniqueName: \"kubernetes.io/projected/504e7b19-a2eb-44f2-88bf-e3324afccb7f-kube-api-access-hqx8l\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:42.928077 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.928038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tmp-dir\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:42.928198 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.928107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-home\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:42.928238 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.928196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-dshm\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:42.928238 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.928217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tls-certs\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:42.928314 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:42.928266    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-model-cache\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.029562 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.029449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqx8l\" (UniqueName: \"kubernetes.io/projected/504e7b19-a2eb-44f2-88bf-e3324afccb7f-kube-api-access-hqx8l\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.029562 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.029525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tmp-dir\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.029562 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.029556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-home\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.029885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.029626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-dshm\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.029885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.029651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tls-certs\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.029885 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.029676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-model-cache\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.030114 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.030045    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tmp-dir\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.030179 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.030120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-home\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.030179 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.030125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-model-cache\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.032328 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.032304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-dshm\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.032687 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.032665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tls-certs\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.038304 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.038279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqx8l\" (UniqueName: \"kubernetes.io/projected/504e7b19-a2eb-44f2-88bf-e3324afccb7f-kube-api-access-hqx8l\") pod \"auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") " pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.076034 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.075964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:43.263250 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.263222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"]
Jul 27 16:59:43.266115 ip-10-0-129-154 kubenswrapper[2579]: W0727 16:59:43.266072    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod504e7b19_a2eb_44f2_88bf_e3324afccb7f.slice/crio-e6833b372c2874c8d9e1f81a8993278ed2ab3bb7154a94155396b4e0447b1102 WatchSource:0}: Error finding container e6833b372c2874c8d9e1f81a8993278ed2ab3bb7154a94155396b4e0447b1102: Status 404 returned error can't find the container with id e6833b372c2874c8d9e1f81a8993278ed2ab3bb7154a94155396b4e0447b1102
Jul 27 16:59:43.791626 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.791583    2579 patch_prober.go:28] interesting pod/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 27 16:59:43.792119 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.791654    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 27 16:59:43.975670 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.975626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc" event={"ID":"504e7b19-a2eb-44f2-88bf-e3324afccb7f","Type":"ContainerStarted","Data":"9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff"}
Jul 27 16:59:43.975870 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.975739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc" event={"ID":"504e7b19-a2eb-44f2-88bf-e3324afccb7f","Type":"ContainerStarted","Data":"e6833b372c2874c8d9e1f81a8993278ed2ab3bb7154a94155396b4e0447b1102"}
Jul 27 16:59:43.994767 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:43.994706    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc" podStartSLOduration=1.9946876169999999 podStartE2EDuration="1.994687617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:59:43.992071516 +0000 UTC m=+2561.617252469" watchObservedRunningTime="2026-07-27 16:59:43.994687617 +0000 UTC m=+2561.619868572"
Jul 27 16:59:44.790774 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:44.790737    2579 patch_prober.go:28] interesting pod/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Jul 27 16:59:44.790988 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:44.790796    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Jul 27 16:59:45.794738 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:45.794705    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:45.795219 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:45.795199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:45.796219 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:45.796193    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 16:59:47.412157 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:47.412114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 16:59:53.076358 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:53.076316    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:53.076800 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:53.076483    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:53.089130 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:53.089101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 16:59:54.033347 ip-10-0-129-154 kubenswrapper[2579]: I0727 16:59:54.033320    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 17:00:12.982754 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:12.982715    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"]
Jul 27 17:00:12.986101 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:12.986079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:12.993336 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:12.993308    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"openshift-service-ca.crt\""
Jul 27 17:00:12.993488 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:12.993415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"auth-test-model-d4529a-kserve-self-signed-certs\""
Jul 27 17:00:12.994427 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:12.994405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"default-dockercfg-nh66d\""
Jul 27 17:00:12.994525 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:12.994435    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"kube-root-ca.crt\""
Jul 27 17:00:13.004978 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.004939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"]
Jul 27 17:00:13.080035 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.079969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-home\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.080276 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.080091    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81732a3e-c145-4012-831e-0ff5ecd0473e-tls-certs\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.080276 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.080120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-model-cache\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.080276 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.080148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drv8h\" (UniqueName: \"kubernetes.io/projected/81732a3e-c145-4012-831e-0ff5ecd0473e-kube-api-access-drv8h\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.080276 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.080222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-dshm\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.080504 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.080361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-tmp-dir\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182021 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.181962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-home\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182203 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81732a3e-c145-4012-831e-0ff5ecd0473e-tls-certs\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182203 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-model-cache\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182203 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drv8h\" (UniqueName: \"kubernetes.io/projected/81732a3e-c145-4012-831e-0ff5ecd0473e-kube-api-access-drv8h\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182203 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-dshm\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182203 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-tmp-dir\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182478 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-home\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182539 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-model-cache\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.182620 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.182604    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-tmp-dir\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.184539 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.184514    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-dshm\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.184854 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.184837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81732a3e-c145-4012-831e-0ff5ecd0473e-tls-certs\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.217644 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.217614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drv8h\" (UniqueName: \"kubernetes.io/projected/81732a3e-c145-4012-831e-0ff5ecd0473e-kube-api-access-drv8h\") pod \"auth-test-model-d4529a-kserve-7756d6f9f-c28ls\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") " pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.296288 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.296187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:13.445695 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:13.445526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"]
Jul 27 17:00:13.448664 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:00:13.448631    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81732a3e_c145_4012_831e_0ff5ecd0473e.slice/crio-8e0b12ab14e904c207c3867a6daa70af00b7ca956d22bb0c866ed760379dda35 WatchSource:0}: Error finding container 8e0b12ab14e904c207c3867a6daa70af00b7ca956d22bb0c866ed760379dda35: Status 404 returned error can't find the container with id 8e0b12ab14e904c207c3867a6daa70af00b7ca956d22bb0c866ed760379dda35
Jul 27 17:00:14.124401 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:14.124365    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls" event={"ID":"81732a3e-c145-4012-831e-0ff5ecd0473e","Type":"ContainerStarted","Data":"9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a"}
Jul 27 17:00:14.124401 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:14.124409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls" event={"ID":"81732a3e-c145-4012-831e-0ff5ecd0473e","Type":"ContainerStarted","Data":"8e0b12ab14e904c207c3867a6daa70af00b7ca956d22bb0c866ed760379dda35"}
Jul 27 17:00:14.142864 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:14.142804    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls" podStartSLOduration=2.142784188 podStartE2EDuration="2.142784188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:00:14.141797056 +0000 UTC m=+2591.766978012" watchObservedRunningTime="2026-07-27 17:00:14.142784188 +0000 UTC m=+2591.767965140"
Jul 27 17:00:16.652350 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:16.652308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:00:23.297137 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:23.297030    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:23.297137 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:23.297093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:23.309692 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:23.309665    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:24.189973 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:24.189940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:00:40.974187 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:00:40.974144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:01:01.955622 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:01:01.955582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:01:21.055836 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:01:21.055795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:01:44.001590 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:01:44.001555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:02:01.658391 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:02:01.658352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:02:02.976554 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:02:02.976525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:02:02.980477 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:02:02.980450    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:02:27.066709 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:02:27.066668    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:02:45.457822 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:02:45.457772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:03:07.446754 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:03:07.446712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:03:21.185571 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:03:21.185531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:03:33.017479 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:03:33.017373    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:03:51.652741 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:03:51.652698    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:04:08.852738 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:08.852697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:04:34.092461 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:34.092420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:04:39.462887 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:39.462851    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"]
Jul 27 17:04:40.435904 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:40.435866    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"]
Jul 27 17:04:48.593634 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.593591    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"]
Jul 27 17:04:48.597218 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.597196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.600096 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.600072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f1b699\"/\"sub-test-model-f1b699-kserve-self-signed-certs\""
Jul 27 17:04:48.608403 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.608367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"]
Jul 27 17:04:48.718471 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.718428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-home\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.718689 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.718483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8kf\" (UniqueName: \"kubernetes.io/projected/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-kube-api-access-lv8kf\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.718689 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.718567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tmp-dir\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.718689 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.718603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-model-cache\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.718853 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.718745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-dshm\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.718853 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.718777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tls-certs\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.819871 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.819818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-dshm\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.819871 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.819873    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tls-certs\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820210 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.819941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-home\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820210 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.819963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8kf\" (UniqueName: \"kubernetes.io/projected/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-kube-api-access-lv8kf\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820210 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.820044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tmp-dir\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820210 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.820075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-model-cache\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820421 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.820380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-home\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820486 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.820425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tmp-dir\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.820543 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.820499    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-model-cache\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.822485 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.822460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-dshm\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.822685 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.822665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tls-certs\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.846749 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.846670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8kf\" (UniqueName: \"kubernetes.io/projected/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-kube-api-access-lv8kf\") pod \"sub-test-model-f1b699-kserve-64bcf96b57-2hctr\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") " pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:48.907698 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:48.907639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:49.059050 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:49.059018    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"]
Jul 27 17:04:49.063205 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:04:49.063178    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee285e78_dee5_48f4_b6d1_ca3c1b5100ae.slice/crio-23928f9e13cc755474f533475d8d02dccd4561a17866975ba2d7710108348665 WatchSource:0}: Error finding container 23928f9e13cc755474f533475d8d02dccd4561a17866975ba2d7710108348665: Status 404 returned error can't find the container with id 23928f9e13cc755474f533475d8d02dccd4561a17866975ba2d7710108348665
Jul 27 17:04:49.475866 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:49.475828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr" event={"ID":"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae","Type":"ContainerStarted","Data":"b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7"}
Jul 27 17:04:49.476118 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:49.475876    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr" event={"ID":"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae","Type":"ContainerStarted","Data":"23928f9e13cc755474f533475d8d02dccd4561a17866975ba2d7710108348665"}
Jul 27 17:04:49.498867 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:49.498803    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr" podStartSLOduration=1.498783447 podStartE2EDuration="1.498783447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:04:49.497172625 +0000 UTC m=+2867.122353583" watchObservedRunningTime="2026-07-27 17:04:49.498783447 +0000 UTC m=+2867.123964470"
Jul 27 17:04:51.858184 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:51.858144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:04:54.484080 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.483958    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc" podUID="504e7b19-a2eb-44f2-88bf-e3324afccb7f" containerName="main" containerID="cri-o://9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff" gracePeriod=45
Jul 27 17:04:54.751889 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.751861    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 17:04:54.790509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqx8l\" (UniqueName: \"kubernetes.io/projected/504e7b19-a2eb-44f2-88bf-e3324afccb7f-kube-api-access-hqx8l\") pod \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") "
Jul 27 17:04:54.790702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790559    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tls-certs\") pod \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") "
Jul 27 17:04:54.790702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790596    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-dshm\") pod \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") "
Jul 27 17:04:54.790702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-home\") pod \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") "
Jul 27 17:04:54.790702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tmp-dir\") pod \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") "
Jul 27 17:04:54.790702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-model-cache\") pod \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\" (UID: \"504e7b19-a2eb-44f2-88bf-e3324afccb7f\") "
Jul 27 17:04:54.790968 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-home" (OuterVolumeSpecName: "home") pod "504e7b19-a2eb-44f2-88bf-e3324afccb7f" (UID: "504e7b19-a2eb-44f2-88bf-e3324afccb7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:54.791044 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.790968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "504e7b19-a2eb-44f2-88bf-e3324afccb7f" (UID: "504e7b19-a2eb-44f2-88bf-e3324afccb7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:54.791149 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.791132    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "504e7b19-a2eb-44f2-88bf-e3324afccb7f" (UID: "504e7b19-a2eb-44f2-88bf-e3324afccb7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:54.793120 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.793082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-dshm" (OuterVolumeSpecName: "dshm") pod "504e7b19-a2eb-44f2-88bf-e3324afccb7f" (UID: "504e7b19-a2eb-44f2-88bf-e3324afccb7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:54.793236 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.793149    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "504e7b19-a2eb-44f2-88bf-e3324afccb7f" (UID: "504e7b19-a2eb-44f2-88bf-e3324afccb7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:54.793236 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.793163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504e7b19-a2eb-44f2-88bf-e3324afccb7f-kube-api-access-hqx8l" (OuterVolumeSpecName: "kube-api-access-hqx8l") pod "504e7b19-a2eb-44f2-88bf-e3324afccb7f" (UID: "504e7b19-a2eb-44f2-88bf-e3324afccb7f"). InnerVolumeSpecName "kube-api-access-hqx8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:04:54.892352 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.892311    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:54.892352 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.892344    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:54.892352 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.892355    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:54.892352 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.892363    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqx8l\" (UniqueName: \"kubernetes.io/projected/504e7b19-a2eb-44f2-88bf-e3324afccb7f-kube-api-access-hqx8l\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:54.892643 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.892373    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/504e7b19-a2eb-44f2-88bf-e3324afccb7f-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:54.892643 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:54.892381    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/504e7b19-a2eb-44f2-88bf-e3324afccb7f-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:55.456642 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.456597    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls" podUID="81732a3e-c145-4012-831e-0ff5ecd0473e" containerName="main" containerID="cri-o://9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a" gracePeriod=45
Jul 27 17:04:55.508323 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.508286    2579 generic.go:356] "Generic (PLEG): container finished" podID="504e7b19-a2eb-44f2-88bf-e3324afccb7f" containerID="9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff" exitCode=0
Jul 27 17:04:55.508773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.508334    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"
Jul 27 17:04:55.508773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.508371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc" event={"ID":"504e7b19-a2eb-44f2-88bf-e3324afccb7f","Type":"ContainerDied","Data":"9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff"}
Jul 27 17:04:55.508773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.508422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc" event={"ID":"504e7b19-a2eb-44f2-88bf-e3324afccb7f","Type":"ContainerDied","Data":"e6833b372c2874c8d9e1f81a8993278ed2ab3bb7154a94155396b4e0447b1102"}
Jul 27 17:04:55.508773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.508445    2579 scope.go:117] "RemoveContainer" containerID="9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff"
Jul 27 17:04:55.530739 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.530703    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"]
Jul 27 17:04:55.536740 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.536708    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/auth-test-model-f1b699-kserve-65bbf4cfc9-t4bbc"]
Jul 27 17:04:55.569083 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.569054    2579 scope.go:117] "RemoveContainer" containerID="9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff"
Jul 27 17:04:55.572150 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:04:55.572109    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff\": container with ID starting with 9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff not found: ID does not exist" containerID="9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff"
Jul 27 17:04:55.572254 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.572151    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff"} err="failed to get container status \"9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff\": rpc error: code = NotFound desc = could not find container \"9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff\": container with ID starting with 9df97f3fa8e09a7eb01751dfe4c19405530b256d242578c447e9afe21dda54ff not found: ID does not exist"
Jul 27 17:04:55.725689 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.725657    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:04:55.801402 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drv8h\" (UniqueName: \"kubernetes.io/projected/81732a3e-c145-4012-831e-0ff5ecd0473e-kube-api-access-drv8h\") pod \"81732a3e-c145-4012-831e-0ff5ecd0473e\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") "
Jul 27 17:04:55.801402 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-model-cache\") pod \"81732a3e-c145-4012-831e-0ff5ecd0473e\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") "
Jul 27 17:04:55.801645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-dshm\") pod \"81732a3e-c145-4012-831e-0ff5ecd0473e\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") "
Jul 27 17:04:55.801645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801460    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-home\") pod \"81732a3e-c145-4012-831e-0ff5ecd0473e\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") "
Jul 27 17:04:55.801645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801497    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81732a3e-c145-4012-831e-0ff5ecd0473e-tls-certs\") pod \"81732a3e-c145-4012-831e-0ff5ecd0473e\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") "
Jul 27 17:04:55.801645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-tmp-dir\") pod \"81732a3e-c145-4012-831e-0ff5ecd0473e\" (UID: \"81732a3e-c145-4012-831e-0ff5ecd0473e\") "
Jul 27 17:04:55.801868 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-model-cache" (OuterVolumeSpecName: "model-cache") pod "81732a3e-c145-4012-831e-0ff5ecd0473e" (UID: "81732a3e-c145-4012-831e-0ff5ecd0473e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:55.801868 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-home" (OuterVolumeSpecName: "home") pod "81732a3e-c145-4012-831e-0ff5ecd0473e" (UID: "81732a3e-c145-4012-831e-0ff5ecd0473e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:55.801987 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801955    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:55.801987 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801981    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:55.802107 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.801992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81732a3e-c145-4012-831e-0ff5ecd0473e" (UID: "81732a3e-c145-4012-831e-0ff5ecd0473e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:55.804039 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.803992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81732a3e-c145-4012-831e-0ff5ecd0473e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81732a3e-c145-4012-831e-0ff5ecd0473e" (UID: "81732a3e-c145-4012-831e-0ff5ecd0473e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:04:55.804039 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.804015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-dshm" (OuterVolumeSpecName: "dshm") pod "81732a3e-c145-4012-831e-0ff5ecd0473e" (UID: "81732a3e-c145-4012-831e-0ff5ecd0473e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:04:55.804190 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.804040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81732a3e-c145-4012-831e-0ff5ecd0473e-kube-api-access-drv8h" (OuterVolumeSpecName: "kube-api-access-drv8h") pod "81732a3e-c145-4012-831e-0ff5ecd0473e" (UID: "81732a3e-c145-4012-831e-0ff5ecd0473e"). InnerVolumeSpecName "kube-api-access-drv8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:04:55.902787 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.902745    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:55.902787 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.902783    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drv8h\" (UniqueName: \"kubernetes.io/projected/81732a3e-c145-4012-831e-0ff5ecd0473e-kube-api-access-drv8h\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:55.902787 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.902792    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81732a3e-c145-4012-831e-0ff5ecd0473e-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:55.902787 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:55.902800    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81732a3e-c145-4012-831e-0ff5ecd0473e-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:04:56.514627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.514582    2579 generic.go:356] "Generic (PLEG): container finished" podID="81732a3e-c145-4012-831e-0ff5ecd0473e" containerID="9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a" exitCode=0
Jul 27 17:04:56.514627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.514624    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"
Jul 27 17:04:56.515186 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.514669    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls" event={"ID":"81732a3e-c145-4012-831e-0ff5ecd0473e","Type":"ContainerDied","Data":"9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a"}
Jul 27 17:04:56.515186 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.514714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls" event={"ID":"81732a3e-c145-4012-831e-0ff5ecd0473e","Type":"ContainerDied","Data":"8e0b12ab14e904c207c3867a6daa70af00b7ca956d22bb0c866ed760379dda35"}
Jul 27 17:04:56.515186 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.514738    2579 scope.go:117] "RemoveContainer" containerID="9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a"
Jul 27 17:04:56.532387 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.532363    2579 scope.go:117] "RemoveContainer" containerID="9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a"
Jul 27 17:04:56.532733 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:04:56.532706    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a\": container with ID starting with 9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a not found: ID does not exist" containerID="9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a"
Jul 27 17:04:56.532841 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.532738    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a"} err="failed to get container status \"9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a\": rpc error: code = NotFound desc = could not find container \"9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a\": container with ID starting with 9f9523c88964468196a727bd07c30ca5f2c7549eb78595777f5739a5ea7d738a not found: ID does not exist"
Jul 27 17:04:56.557233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.557188    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"]
Jul 27 17:04:56.563604 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.563572    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/auth-test-model-d4529a-kserve-7756d6f9f-c28ls"]
Jul 27 17:04:56.968136 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.968099    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504e7b19-a2eb-44f2-88bf-e3324afccb7f" path="/var/lib/kubelet/pods/504e7b19-a2eb-44f2-88bf-e3324afccb7f/volumes"
Jul 27 17:04:56.968537 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:56.968522    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81732a3e-c145-4012-831e-0ff5ecd0473e" path="/var/lib/kubelet/pods/81732a3e-c145-4012-831e-0ff5ecd0473e/volumes"
Jul 27 17:04:58.908079 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:58.908032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:58.908468 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:58.908095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:58.921129 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:58.921099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:04:59.551814 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:04:59.551781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:05:28.277087 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277043    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"]
Jul 27 17:05:28.277709 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277575    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81732a3e-c145-4012-831e-0ff5ecd0473e" containerName="main"
Jul 27 17:05:28.277709 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277597    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="81732a3e-c145-4012-831e-0ff5ecd0473e" containerName="main"
Jul 27 17:05:28.277709 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277612    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504e7b19-a2eb-44f2-88bf-e3324afccb7f" containerName="main"
Jul 27 17:05:28.277709 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277621    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="504e7b19-a2eb-44f2-88bf-e3324afccb7f" containerName="main"
Jul 27 17:05:28.277921 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277728    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="81732a3e-c145-4012-831e-0ff5ecd0473e" containerName="main"
Jul 27 17:05:28.277921 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.277741    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="504e7b19-a2eb-44f2-88bf-e3324afccb7f" containerName="main"
Jul 27 17:05:28.280971 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.280947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.284891 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.284851    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"sub-test-model-d4529a-kserve-self-signed-certs\""
Jul 27 17:05:28.284891 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.284851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"openshift-service-ca.crt\""
Jul 27 17:05:28.285157 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.284939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"default-dockercfg-nh66d\""
Jul 27 17:05:28.285157 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.284859    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"kube-root-ca.crt\""
Jul 27 17:05:28.293364 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.293309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"]
Jul 27 17:05:28.330963 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.330929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-dshm\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.331178 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.330970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbpr7\" (UniqueName: \"kubernetes.io/projected/971e3d62-bfda-429a-b522-d80de200dc2f-kube-api-access-zbpr7\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.331178 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.331058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971e3d62-bfda-429a-b522-d80de200dc2f-tls-certs\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.331178 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.331111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-model-cache\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.331178 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.331138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-tmp-dir\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.331178 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.331168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-home\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.431757 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.431715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-home\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.431963 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.431798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-dshm\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.431963 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.431823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbpr7\" (UniqueName: \"kubernetes.io/projected/971e3d62-bfda-429a-b522-d80de200dc2f-kube-api-access-zbpr7\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.431963 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.431857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971e3d62-bfda-429a-b522-d80de200dc2f-tls-certs\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.431963 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.431881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-model-cache\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.431963 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.431921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-tmp-dir\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.432259 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.432223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-home\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.432312 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.432293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-model-cache\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.432381 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.432357    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-tmp-dir\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.434423 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.434400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-dshm\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.434698 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.434681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971e3d62-bfda-429a-b522-d80de200dc2f-tls-certs\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.445169 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.445134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbpr7\" (UniqueName: \"kubernetes.io/projected/971e3d62-bfda-429a-b522-d80de200dc2f-kube-api-access-zbpr7\") pod \"sub-test-model-d4529a-kserve-5cf796d877-ftklf\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") " pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.592096 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.591967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:28.748816 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:28.748787    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"]
Jul 27 17:05:28.751102 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:05:28.751069    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod971e3d62_bfda_429a_b522_d80de200dc2f.slice/crio-f68042405522c34897c96004a8e054592de13af348c64932265dbb584415dfc6 WatchSource:0}: Error finding container f68042405522c34897c96004a8e054592de13af348c64932265dbb584415dfc6: Status 404 returned error can't find the container with id f68042405522c34897c96004a8e054592de13af348c64932265dbb584415dfc6
Jul 27 17:05:29.691505 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:29.691447    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf" event={"ID":"971e3d62-bfda-429a-b522-d80de200dc2f","Type":"ContainerStarted","Data":"0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c"}
Jul 27 17:05:29.691505 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:29.691504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf" event={"ID":"971e3d62-bfda-429a-b522-d80de200dc2f","Type":"ContainerStarted","Data":"f68042405522c34897c96004a8e054592de13af348c64932265dbb584415dfc6"}
Jul 27 17:05:29.712448 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:29.712381    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf" podStartSLOduration=1.712360771 podStartE2EDuration="1.712360771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:05:29.710621688 +0000 UTC m=+2907.335802641" watchObservedRunningTime="2026-07-27 17:05:29.712360771 +0000 UTC m=+2907.337541723"
Jul 27 17:05:32.280464 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:32.280413    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:05:38.592376 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:38.592311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:38.592903 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:38.592399    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:38.605285 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:38.605251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:05:38.746958 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:05:38.746917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:06:04.672688 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:04.672649    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:06:23.285650 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:23.285556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:06:28.292197 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:28.292154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"]
Jul 27 17:06:29.188780 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:29.188738    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"]
Jul 27 17:06:35.901608 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.901568    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"]
Jul 27 17:06:35.905430 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.905402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:35.908605 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.908577    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f1b699\"/\"rate-test-model-f1b699-kserve-self-signed-certs\""
Jul 27 17:06:35.917234 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.917202    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"]
Jul 27 17:06:35.931514 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.931480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-home\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:35.931752 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.931538    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7lp\" (UniqueName: \"kubernetes.io/projected/9d4d3506-619b-4dac-9fc6-3b825c375b29-kube-api-access-gs7lp\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:35.931752 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.931604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-model-cache\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:35.931752 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.931636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4d3506-619b-4dac-9fc6-3b825c375b29-tls-certs\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:35.931752 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.931677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-tmp-dir\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:35.931752 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:35.931735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-dshm\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033129 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033086    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7lp\" (UniqueName: \"kubernetes.io/projected/9d4d3506-619b-4dac-9fc6-3b825c375b29-kube-api-access-gs7lp\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033316 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-model-cache\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033389 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4d3506-619b-4dac-9fc6-3b825c375b29-tls-certs\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033389 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-tmp-dir\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033494 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-dshm\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033494 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033470    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-model-cache\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033600 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-home\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.033780 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.033751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-tmp-dir\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.034722 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.034693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-home\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.036550 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.036526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4d3506-619b-4dac-9fc6-3b825c375b29-tls-certs\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.036688 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.036662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-dshm\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.044649 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.044614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7lp\" (UniqueName: \"kubernetes.io/projected/9d4d3506-619b-4dac-9fc6-3b825c375b29-kube-api-access-gs7lp\") pod \"rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") " pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.215400 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.215353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:36.378327 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:36.378288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"]
Jul 27 17:06:36.381563 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:06:36.381535    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d4d3506_619b_4dac_9fc6_3b825c375b29.slice/crio-8866da5c6c07fec531c205ae3746168d61323f265b28234d2db3a9e45e5715f8 WatchSource:0}: Error finding container 8866da5c6c07fec531c205ae3746168d61323f265b28234d2db3a9e45e5715f8: Status 404 returned error can't find the container with id 8866da5c6c07fec531c205ae3746168d61323f265b28234d2db3a9e45e5715f8
Jul 27 17:06:37.028225 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:37.028169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn" event={"ID":"9d4d3506-619b-4dac-9fc6-3b825c375b29","Type":"ContainerStarted","Data":"aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f"}
Jul 27 17:06:37.028225 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:37.028229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn" event={"ID":"9d4d3506-619b-4dac-9fc6-3b825c375b29","Type":"ContainerStarted","Data":"8866da5c6c07fec531c205ae3746168d61323f265b28234d2db3a9e45e5715f8"}
Jul 27 17:06:37.051978 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:37.051918    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn" podStartSLOduration=2.051898118 podStartE2EDuration="2.051898118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:37.049096661 +0000 UTC m=+2974.674277616" watchObservedRunningTime="2026-07-27 17:06:37.051898118 +0000 UTC m=+2974.677079070"
Jul 27 17:06:41.271754 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:41.271713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:06:43.312396 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.312331    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr" podUID="ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" containerName="main" containerID="cri-o://b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7" gracePeriod=45
Jul 27 17:06:43.583377 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.583348    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:06:43.604402 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-dshm\") pod \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") "
Jul 27 17:06:43.604611 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604424    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tmp-dir\") pod \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") "
Jul 27 17:06:43.604611 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-home\") pod \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") "
Jul 27 17:06:43.604611 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-model-cache\") pod \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") "
Jul 27 17:06:43.604611 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tls-certs\") pod \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") "
Jul 27 17:06:43.604811 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv8kf\" (UniqueName: \"kubernetes.io/projected/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-kube-api-access-lv8kf\") pod \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\" (UID: \"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae\") "
Jul 27 17:06:43.604811 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604765    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-home" (OuterVolumeSpecName: "home") pod "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" (UID: "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:43.604811 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" (UID: "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:43.604972 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.604773    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" (UID: "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:43.605067 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.605044    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:43.605120 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.605071    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:43.605120 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.605088    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:43.606892 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.606865    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-dshm" (OuterVolumeSpecName: "dshm") pod "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" (UID: "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:43.607076 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.606930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" (UID: "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:06:43.607076 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.606947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-kube-api-access-lv8kf" (OuterVolumeSpecName: "kube-api-access-lv8kf") pod "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" (UID: "ee285e78-dee5-48f4-b6d1-ca3c1b5100ae"). InnerVolumeSpecName "kube-api-access-lv8kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:06:43.705975 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.705935    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:43.705975 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.705968    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:43.705975 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:43.705980    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8kf\" (UniqueName: \"kubernetes.io/projected/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae-kube-api-access-lv8kf\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.063903 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.063863    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" containerID="b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7" exitCode=0
Jul 27 17:06:44.063903 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.063907    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"
Jul 27 17:06:44.064216 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.063951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr" event={"ID":"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae","Type":"ContainerDied","Data":"b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7"}
Jul 27 17:06:44.064216 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.063994    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr" event={"ID":"ee285e78-dee5-48f4-b6d1-ca3c1b5100ae","Type":"ContainerDied","Data":"23928f9e13cc755474f533475d8d02dccd4561a17866975ba2d7710108348665"}
Jul 27 17:06:44.064216 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.064039    2579 scope.go:117] "RemoveContainer" containerID="b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7"
Jul 27 17:06:44.073567 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.073549    2579 scope.go:117] "RemoveContainer" containerID="b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7"
Jul 27 17:06:44.073866 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:06:44.073847    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7\": container with ID starting with b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7 not found: ID does not exist" containerID="b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7"
Jul 27 17:06:44.073918 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.073876    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7"} err="failed to get container status \"b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7\": rpc error: code = NotFound desc = could not find container \"b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7\": container with ID starting with b38bedeb8003914ad3a90089ed6bc554c077080125bcea0e5db8a44b8cf0f2e7 not found: ID does not exist"
Jul 27 17:06:44.093233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.093196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"]
Jul 27 17:06:44.096581 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.096536    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/sub-test-model-f1b699-kserve-64bcf96b57-2hctr"]
Jul 27 17:06:44.208995 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.208953    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf" podUID="971e3d62-bfda-429a-b522-d80de200dc2f" containerName="main" containerID="cri-o://0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c" gracePeriod=45
Jul 27 17:06:44.474833 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.474801    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:06:44.513509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbpr7\" (UniqueName: \"kubernetes.io/projected/971e3d62-bfda-429a-b522-d80de200dc2f-kube-api-access-zbpr7\") pod \"971e3d62-bfda-429a-b522-d80de200dc2f\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") "
Jul 27 17:06:44.513675 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513517    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-home\") pod \"971e3d62-bfda-429a-b522-d80de200dc2f\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") "
Jul 27 17:06:44.513675 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-dshm\") pod \"971e3d62-bfda-429a-b522-d80de200dc2f\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") "
Jul 27 17:06:44.513675 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513605    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971e3d62-bfda-429a-b522-d80de200dc2f-tls-certs\") pod \"971e3d62-bfda-429a-b522-d80de200dc2f\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") "
Jul 27 17:06:44.513675 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-tmp-dir\") pod \"971e3d62-bfda-429a-b522-d80de200dc2f\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") "
Jul 27 17:06:44.513908 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-model-cache\") pod \"971e3d62-bfda-429a-b522-d80de200dc2f\" (UID: \"971e3d62-bfda-429a-b522-d80de200dc2f\") "
Jul 27 17:06:44.513908 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-home" (OuterVolumeSpecName: "home") pod "971e3d62-bfda-429a-b522-d80de200dc2f" (UID: "971e3d62-bfda-429a-b522-d80de200dc2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:44.514047 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.513970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "971e3d62-bfda-429a-b522-d80de200dc2f" (UID: "971e3d62-bfda-429a-b522-d80de200dc2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:44.514118 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.514086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "971e3d62-bfda-429a-b522-d80de200dc2f" (UID: "971e3d62-bfda-429a-b522-d80de200dc2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:44.514223 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.514207    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.514264 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.514230    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.514264 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.514243    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.516066 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.516027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971e3d62-bfda-429a-b522-d80de200dc2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "971e3d62-bfda-429a-b522-d80de200dc2f" (UID: "971e3d62-bfda-429a-b522-d80de200dc2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:06:44.516324 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.516297    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-dshm" (OuterVolumeSpecName: "dshm") pod "971e3d62-bfda-429a-b522-d80de200dc2f" (UID: "971e3d62-bfda-429a-b522-d80de200dc2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:06:44.516420 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.516379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971e3d62-bfda-429a-b522-d80de200dc2f-kube-api-access-zbpr7" (OuterVolumeSpecName: "kube-api-access-zbpr7") pod "971e3d62-bfda-429a-b522-d80de200dc2f" (UID: "971e3d62-bfda-429a-b522-d80de200dc2f"). InnerVolumeSpecName "kube-api-access-zbpr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:06:44.615404 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.615301    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbpr7\" (UniqueName: \"kubernetes.io/projected/971e3d62-bfda-429a-b522-d80de200dc2f-kube-api-access-zbpr7\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.615404 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.615337    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/971e3d62-bfda-429a-b522-d80de200dc2f-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.615404 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.615347    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/971e3d62-bfda-429a-b522-d80de200dc2f-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:06:44.968718 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:44.968680    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" path="/var/lib/kubelet/pods/ee285e78-dee5-48f4-b6d1-ca3c1b5100ae/volumes"
Jul 27 17:06:45.071753 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.071716    2579 generic.go:356] "Generic (PLEG): container finished" podID="971e3d62-bfda-429a-b522-d80de200dc2f" containerID="0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c" exitCode=0
Jul 27 17:06:45.071958 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.071773    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"
Jul 27 17:06:45.071958 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.071809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf" event={"ID":"971e3d62-bfda-429a-b522-d80de200dc2f","Type":"ContainerDied","Data":"0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c"}
Jul 27 17:06:45.071958 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.071856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf" event={"ID":"971e3d62-bfda-429a-b522-d80de200dc2f","Type":"ContainerDied","Data":"f68042405522c34897c96004a8e054592de13af348c64932265dbb584415dfc6"}
Jul 27 17:06:45.071958 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.071880    2579 scope.go:117] "RemoveContainer" containerID="0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c"
Jul 27 17:06:45.081476 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.081454    2579 scope.go:117] "RemoveContainer" containerID="0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c"
Jul 27 17:06:45.081828 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:06:45.081804    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c\": container with ID starting with 0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c not found: ID does not exist" containerID="0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c"
Jul 27 17:06:45.081886 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.081838    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c"} err="failed to get container status \"0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c\": rpc error: code = NotFound desc = could not find container \"0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c\": container with ID starting with 0e10d89abf80f703f78a774195c74109bd7f83ac709b562049003cfdf45e159c not found: ID does not exist"
Jul 27 17:06:45.102545 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.102503    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"]
Jul 27 17:06:45.106302 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:45.106270    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/sub-test-model-d4529a-kserve-5cf796d877-ftklf"]
Jul 27 17:06:46.216096 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:46.216043    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:46.216096 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:46.216111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:46.229239 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:46.229206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:06:46.968393 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:46.968355    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="971e3d62-bfda-429a-b522-d80de200dc2f" path="/var/lib/kubelet/pods/971e3d62-bfda-429a-b522-d80de200dc2f/volumes"
Jul 27 17:06:47.094416 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:06:47.094379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:07:02.744296 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.744254    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"]
Jul 27 17:07:02.745509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.745480    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="971e3d62-bfda-429a-b522-d80de200dc2f" containerName="main"
Jul 27 17:07:02.745509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.745507    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="971e3d62-bfda-429a-b522-d80de200dc2f" containerName="main"
Jul 27 17:07:02.745509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.745517    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" containerName="main"
Jul 27 17:07:02.745773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.745523    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" containerName="main"
Jul 27 17:07:02.745773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.745615    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="971e3d62-bfda-429a-b522-d80de200dc2f" containerName="main"
Jul 27 17:07:02.745773 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.745623    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee285e78-dee5-48f4-b6d1-ca3c1b5100ae" containerName="main"
Jul 27 17:07:02.750379 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.750354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:02.795662 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.795622    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"openshift-service-ca.crt\""
Jul 27 17:07:02.795897 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.795791    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"kube-root-ca.crt\""
Jul 27 17:07:02.795897 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.795792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"default-dockercfg-nh66d\""
Jul 27 17:07:02.796241 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.796226    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d4529a\"/\"rate-test-model-d4529a-kserve-self-signed-certs\""
Jul 27 17:07:02.812906 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.812870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"]
Jul 27 17:07:02.904357 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.904314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-model-cache\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:02.904357 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.904357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-home\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:02.904603 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.904379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fb2572-b045-4461-b868-8a7a3a41fe44-tls-certs\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:02.904603 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.904462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-tmp-dir\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:02.904603 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.904502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphh8\" (UniqueName: \"kubernetes.io/projected/d3fb2572-b045-4461-b868-8a7a3a41fe44-kube-api-access-qphh8\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:02.904603 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:02.904566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-dshm\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005245 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005131    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-model-cache\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005245 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-home\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005245 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fb2572-b045-4461-b868-8a7a3a41fe44-tls-certs\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005567 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-tmp-dir\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005567 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphh8\" (UniqueName: \"kubernetes.io/projected/d3fb2572-b045-4461-b868-8a7a3a41fe44-kube-api-access-qphh8\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005567 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005409    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-dshm\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005739 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-model-cache\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005739 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-home\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.005739 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.005660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-tmp-dir\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.008093 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.008019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-dshm\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.008405 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.008383    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:07:03.008541 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.008427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fb2572-b045-4461-b868-8a7a3a41fe44-tls-certs\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.013939 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.013914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:07:03.023768 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.023737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphh8\" (UniqueName: \"kubernetes.io/projected/d3fb2572-b045-4461-b868-8a7a3a41fe44-kube-api-access-qphh8\") pod \"rate-test-model-d4529a-kserve-768f5f7875-bts6b\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") " pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.060771 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.060731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:03.226194 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:03.226161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"]
Jul 27 17:07:03.229284 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:07:03.229256    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3fb2572_b045_4461_b868_8a7a3a41fe44.slice/crio-334c4e6572def8aed2a7a03e71fe982682a076481791973f6ed403bc004f4def WatchSource:0}: Error finding container 334c4e6572def8aed2a7a03e71fe982682a076481791973f6ed403bc004f4def: Status 404 returned error can't find the container with id 334c4e6572def8aed2a7a03e71fe982682a076481791973f6ed403bc004f4def
Jul 27 17:07:04.168070 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:04.167993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b" event={"ID":"d3fb2572-b045-4461-b868-8a7a3a41fe44","Type":"ContainerStarted","Data":"9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2"}
Jul 27 17:07:04.168489 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:04.168076    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b" event={"ID":"d3fb2572-b045-4461-b868-8a7a3a41fe44","Type":"ContainerStarted","Data":"334c4e6572def8aed2a7a03e71fe982682a076481791973f6ed403bc004f4def"}
Jul 27 17:07:04.214050 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:04.213968    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b" podStartSLOduration=2.213949779 podStartE2EDuration="2.213949779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:04.210989267 +0000 UTC m=+3001.836170233" watchObservedRunningTime="2026-07-27 17:07:04.213949779 +0000 UTC m=+3001.839130732"
Jul 27 17:07:06.280098 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:06.280062    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:07:13.061893 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:13.061850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:13.062514 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:13.061909    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:13.075038 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:13.074987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:13.229993 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:13.229962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:07:38.358455 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:38.358411    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:07:57.663140 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:07:57.663041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:08:04.756082 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:04.756046    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"]
Jul 27 17:08:05.657668 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:05.657630    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"]
Jul 27 17:08:12.466841 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:12.466795    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:08:19.778484 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:19.778436    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn" podUID="9d4d3506-619b-4dac-9fc6-3b825c375b29" containerName="main" containerID="cri-o://aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f" gracePeriod=45
Jul 27 17:08:20.030506 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.030436    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:08:20.084690 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.084654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4d3506-619b-4dac-9fc6-3b825c375b29-tls-certs\") pod \"9d4d3506-619b-4dac-9fc6-3b825c375b29\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") "
Jul 27 17:08:20.084909 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.084719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-dshm\") pod \"9d4d3506-619b-4dac-9fc6-3b825c375b29\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") "
Jul 27 17:08:20.084909 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.084775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-model-cache\") pod \"9d4d3506-619b-4dac-9fc6-3b825c375b29\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") "
Jul 27 17:08:20.084909 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.084803    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs7lp\" (UniqueName: \"kubernetes.io/projected/9d4d3506-619b-4dac-9fc6-3b825c375b29-kube-api-access-gs7lp\") pod \"9d4d3506-619b-4dac-9fc6-3b825c375b29\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") "
Jul 27 17:08:20.085235 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.085192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d4d3506-619b-4dac-9fc6-3b825c375b29" (UID: "9d4d3506-619b-4dac-9fc6-3b825c375b29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.087185 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.087156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-dshm" (OuterVolumeSpecName: "dshm") pod "9d4d3506-619b-4dac-9fc6-3b825c375b29" (UID: "9d4d3506-619b-4dac-9fc6-3b825c375b29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.087456 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.087431    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4d3506-619b-4dac-9fc6-3b825c375b29-kube-api-access-gs7lp" (OuterVolumeSpecName: "kube-api-access-gs7lp") pod "9d4d3506-619b-4dac-9fc6-3b825c375b29" (UID: "9d4d3506-619b-4dac-9fc6-3b825c375b29"). InnerVolumeSpecName "kube-api-access-gs7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:08:20.087600 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.087570    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4d3506-619b-4dac-9fc6-3b825c375b29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d4d3506-619b-4dac-9fc6-3b825c375b29" (UID: "9d4d3506-619b-4dac-9fc6-3b825c375b29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:08:20.186229 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-home\") pod \"9d4d3506-619b-4dac-9fc6-3b825c375b29\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") "
Jul 27 17:08:20.186229 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-tmp-dir\") pod \"9d4d3506-619b-4dac-9fc6-3b825c375b29\" (UID: \"9d4d3506-619b-4dac-9fc6-3b825c375b29\") "
Jul 27 17:08:20.186476 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186463    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.186517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186478    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.186517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-home" (OuterVolumeSpecName: "home") pod "9d4d3506-619b-4dac-9fc6-3b825c375b29" (UID: "9d4d3506-619b-4dac-9fc6-3b825c375b29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.186517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186488    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7lp\" (UniqueName: \"kubernetes.io/projected/9d4d3506-619b-4dac-9fc6-3b825c375b29-kube-api-access-gs7lp\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.186609 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186524    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d4d3506-619b-4dac-9fc6-3b825c375b29-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.186609 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.186521    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d4d3506-619b-4dac-9fc6-3b825c375b29" (UID: "9d4d3506-619b-4dac-9fc6-3b825c375b29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.288121 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.287990    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.288121 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.288055    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d4d3506-619b-4dac-9fc6-3b825c375b29-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.556858 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.556757    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d4d3506-619b-4dac-9fc6-3b825c375b29" containerID="aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f" exitCode=0
Jul 27 17:08:20.556858 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.556798    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"
Jul 27 17:08:20.556858 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.556829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn" event={"ID":"9d4d3506-619b-4dac-9fc6-3b825c375b29","Type":"ContainerDied","Data":"aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f"}
Jul 27 17:08:20.557188 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.556862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn" event={"ID":"9d4d3506-619b-4dac-9fc6-3b825c375b29","Type":"ContainerDied","Data":"8866da5c6c07fec531c205ae3746168d61323f265b28234d2db3a9e45e5715f8"}
Jul 27 17:08:20.557188 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.556882    2579 scope.go:117] "RemoveContainer" containerID="aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f"
Jul 27 17:08:20.566553 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.566534    2579 scope.go:117] "RemoveContainer" containerID="aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f"
Jul 27 17:08:20.566911 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:08:20.566891    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f\": container with ID starting with aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f not found: ID does not exist" containerID="aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f"
Jul 27 17:08:20.566961 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.566922    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f"} err="failed to get container status \"aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f\": rpc error: code = NotFound desc = could not find container \"aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f\": container with ID starting with aea04d7519973ce5d7b397628181bfd9f7eacb4168a926236c52df441639ca5f not found: ID does not exist"
Jul 27 17:08:20.582340 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.582298    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"]
Jul 27 17:08:20.586754 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.586720    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f1b699/rate-test-model-f1b699-kserve-59d4d6c4d7-4rnzn"]
Jul 27 17:08:20.678359 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.678289    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b" podUID="d3fb2572-b045-4461-b868-8a7a3a41fe44" containerName="main" containerID="cri-o://9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2" gracePeriod=45
Jul 27 17:08:20.928460 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.928427    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:08:20.968239 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.968194    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d4d3506-619b-4dac-9fc6-3b825c375b29" path="/var/lib/kubelet/pods/9d4d3506-619b-4dac-9fc6-3b825c375b29/volumes"
Jul 27 17:08:20.994333 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994294    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fb2572-b045-4461-b868-8a7a3a41fe44-tls-certs\") pod \"d3fb2572-b045-4461-b868-8a7a3a41fe44\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") "
Jul 27 17:08:20.994541 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qphh8\" (UniqueName: \"kubernetes.io/projected/d3fb2572-b045-4461-b868-8a7a3a41fe44-kube-api-access-qphh8\") pod \"d3fb2572-b045-4461-b868-8a7a3a41fe44\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") "
Jul 27 17:08:20.994541 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-home\") pod \"d3fb2572-b045-4461-b868-8a7a3a41fe44\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") "
Jul 27 17:08:20.994541 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-model-cache\") pod \"d3fb2572-b045-4461-b868-8a7a3a41fe44\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") "
Jul 27 17:08:20.994541 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-tmp-dir\") pod \"d3fb2572-b045-4461-b868-8a7a3a41fe44\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") "
Jul 27 17:08:20.994541 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-dshm\") pod \"d3fb2572-b045-4461-b868-8a7a3a41fe44\" (UID: \"d3fb2572-b045-4461-b868-8a7a3a41fe44\") "
Jul 27 17:08:20.994799 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-home" (OuterVolumeSpecName: "home") pod "d3fb2572-b045-4461-b868-8a7a3a41fe44" (UID: "d3fb2572-b045-4461-b868-8a7a3a41fe44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.994799 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994777    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3fb2572-b045-4461-b868-8a7a3a41fe44" (UID: "d3fb2572-b045-4461-b868-8a7a3a41fe44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.994920 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994877    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.994920 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.994896    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:20.995431 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.995402    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3fb2572-b045-4461-b868-8a7a3a41fe44" (UID: "d3fb2572-b045-4461-b868-8a7a3a41fe44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.997019 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.996984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fb2572-b045-4461-b868-8a7a3a41fe44-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3fb2572-b045-4461-b868-8a7a3a41fe44" (UID: "d3fb2572-b045-4461-b868-8a7a3a41fe44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:08:20.997119 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.997030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-dshm" (OuterVolumeSpecName: "dshm") pod "d3fb2572-b045-4461-b868-8a7a3a41fe44" (UID: "d3fb2572-b045-4461-b868-8a7a3a41fe44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:08:20.997119 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:20.997044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fb2572-b045-4461-b868-8a7a3a41fe44-kube-api-access-qphh8" (OuterVolumeSpecName: "kube-api-access-qphh8") pod "d3fb2572-b045-4461-b868-8a7a3a41fe44" (UID: "d3fb2572-b045-4461-b868-8a7a3a41fe44"). InnerVolumeSpecName "kube-api-access-qphh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:08:21.095742 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.095690    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:21.095742 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.095732    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3fb2572-b045-4461-b868-8a7a3a41fe44-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:21.095742 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.095744    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3fb2572-b045-4461-b868-8a7a3a41fe44-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:21.095742 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.095754    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphh8\" (UniqueName: \"kubernetes.io/projected/d3fb2572-b045-4461-b868-8a7a3a41fe44-kube-api-access-qphh8\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:08:21.563956 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.563913    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3fb2572-b045-4461-b868-8a7a3a41fe44" containerID="9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2" exitCode=0
Jul 27 17:08:21.563956 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.563961    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"
Jul 27 17:08:21.564223 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.563977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b" event={"ID":"d3fb2572-b045-4461-b868-8a7a3a41fe44","Type":"ContainerDied","Data":"9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2"}
Jul 27 17:08:21.564223 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.564042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b" event={"ID":"d3fb2572-b045-4461-b868-8a7a3a41fe44","Type":"ContainerDied","Data":"334c4e6572def8aed2a7a03e71fe982682a076481791973f6ed403bc004f4def"}
Jul 27 17:08:21.564223 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.564070    2579 scope.go:117] "RemoveContainer" containerID="9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2"
Jul 27 17:08:21.573301 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.573277    2579 scope.go:117] "RemoveContainer" containerID="9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2"
Jul 27 17:08:21.573638 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:08:21.573618    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2\": container with ID starting with 9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2 not found: ID does not exist" containerID="9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2"
Jul 27 17:08:21.573682 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.573648    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2"} err="failed to get container status \"9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2\": rpc error: code = NotFound desc = could not find container \"9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2\": container with ID starting with 9163b5c198b028ab56edac286ca09afd52ae3aae272640fbf04b6c6bafcdcfd2 not found: ID does not exist"
Jul 27 17:08:21.586959 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.586918    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"]
Jul 27 17:08:21.592549 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:21.592519    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d4529a/rate-test-model-d4529a-kserve-768f5f7875-bts6b"]
Jul 27 17:08:22.968607 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:22.968551    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3fb2572-b045-4461-b868-8a7a3a41fe44" path="/var/lib/kubelet/pods/d3fb2572-b045-4461-b868-8a7a3a41fe44/volumes"
Jul 27 17:08:25.764097 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764056    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"]
Jul 27 17:08:25.764517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764422    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d4d3506-619b-4dac-9fc6-3b825c375b29" containerName="main"
Jul 27 17:08:25.764517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764433    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d4d3506-619b-4dac-9fc6-3b825c375b29" containerName="main"
Jul 27 17:08:25.764517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764473    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3fb2572-b045-4461-b868-8a7a3a41fe44" containerName="main"
Jul 27 17:08:25.764517 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764479    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3fb2572-b045-4461-b868-8a7a3a41fe44" containerName="main"
Jul 27 17:08:25.764661 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764539    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d4d3506-619b-4dac-9fc6-3b825c375b29" containerName="main"
Jul 27 17:08:25.764661 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.764548    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3fb2572-b045-4461-b868-8a7a3a41fe44" containerName="main"
Jul 27 17:08:25.769284 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.769261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.772110 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.772085    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2cf641-openshift-default-dockercfg-xm25q\""
Jul 27 17:08:25.780338 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.780305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"]
Jul 27 17:08:25.846455 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-podinfo\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846655 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846479    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-credential-socket\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846655 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istiod-ca-cert\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846655 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-certs\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846655 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-token\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846655 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-socket\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846831 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-envoy\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846831 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-data\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.846831 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.846701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng6cb\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-kube-api-access-ng6cb\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947413 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-certs\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947413 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-token\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947676 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-socket\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947676 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-envoy\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947676 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-data\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947676 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng6cb\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-kube-api-access-ng6cb\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947676 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-podinfo\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947934 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-credential-socket\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947934 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istiod-ca-cert\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.947934 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-certs\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.948151 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.947986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-data\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.948151 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.948056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-socket\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.948151 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.948097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-credential-socket\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.948593 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.948554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istiod-ca-cert\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.950076 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.950058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-envoy\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.950524 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.950505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-podinfo\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.957784 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.957744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-token\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:25.958082 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:25.958062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng6cb\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-kube-api-access-ng6cb\") pod \"e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") " pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:26.081876 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.081676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:26.236174 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.236070    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"]
Jul 27 17:08:26.239030 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:08:26.238984    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3b6392_9e15_4b36_8ee1_b71bf781190a.slice/crio-608f95a4b1884ecd41193d553cb19d4ba7d9417b0fdc0e1f3676f64d61592aeb WatchSource:0}: Error finding container 608f95a4b1884ecd41193d553cb19d4ba7d9417b0fdc0e1f3676f64d61592aeb: Status 404 returned error can't find the container with id 608f95a4b1884ecd41193d553cb19d4ba7d9417b0fdc0e1f3676f64d61592aeb
Jul 27 17:08:26.241411 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.241365    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:08:26.241513 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.241464    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:08:26.241560 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.241509    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:08:26.591765 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.591726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" event={"ID":"8d3b6392-9e15-4b36-8ee1-b71bf781190a","Type":"ContainerStarted","Data":"8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"}
Jul 27 17:08:26.592027 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:26.591795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" event={"ID":"8d3b6392-9e15-4b36-8ee1-b71bf781190a","Type":"ContainerStarted","Data":"608f95a4b1884ecd41193d553cb19d4ba7d9417b0fdc0e1f3676f64d61592aeb"}
Jul 27 17:08:27.082260 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:27.082221    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:27.087714 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:27.087675    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:27.117642 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:27.117580    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podStartSLOduration=2.117561009 podStartE2EDuration="2.117561009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:26.62552624 +0000 UTC m=+3084.250707192" watchObservedRunningTime="2026-07-27 17:08:27.117561009 +0000 UTC m=+3084.742742038"
Jul 27 17:08:27.596330 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:27.596296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:27.597341 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:27.597318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:37.265070 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:37.265029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:08:43.637228 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:43.637179    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 27 17:08:43.637751 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:43.637255    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:08:45.685948 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:45.685844    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerID="8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451" exitCode=0
Jul 27 17:08:45.685948 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:45.685918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" event={"ID":"8d3b6392-9e15-4b36-8ee1-b71bf781190a","Type":"ContainerDied","Data":"8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"}
Jul 27 17:08:45.686448 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:45.686402    2579 scope.go:117] "RemoveContainer" containerID="8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"
Jul 27 17:08:45.687630 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:45.687598    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:08:45.687712 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:45.687676    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:08:45.687751 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:45.687712    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:08:46.693837 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:46.693795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" event={"ID":"8d3b6392-9e15-4b36-8ee1-b71bf781190a","Type":"ContainerStarted","Data":"35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67"}
Jul 27 17:08:47.082475 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:47.082423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:47.082777 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:47.082752    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 17:08:47.082837 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:47.082810    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 17:08:48.082533 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:48.082495    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 17:08:48.082952 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:48.082576    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 17:08:49.082926 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:49.082886    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 17:08:49.083385 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:49.082960    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 17:08:50.082820 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:50.082777    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 17:08:50.083061 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:50.082868    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 17:08:51.082648 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:51.082608    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 17:08:51.082856 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:51.082700    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 17:08:52.082286 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:52.082239    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Jul 27 17:08:52.082704 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:52.082306    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Jul 27 17:08:53.086161 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:53.086129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:53.086635 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:53.086402    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:53.087210 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:53.087192    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:08:56.062296 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:08:56.062256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:09:24.843931 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.843845    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"]
Jul 27 17:09:24.847396 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.847377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:24.850075 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.850049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2cf641\"/\"kube-root-ca.crt\""
Jul 27 17:09:24.851113 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.851020    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2cf641\"/\"ipp-route-2cf641-kserve-self-signed-certs\""
Jul 27 17:09:24.851113 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.851083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2cf641\"/\"default-dockercfg-5fcrr\""
Jul 27 17:09:24.851348 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.851239    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2cf641\"/\"openshift-service-ca.crt\""
Jul 27 17:09:24.858851 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.858821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"]
Jul 27 17:09:24.968605 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.968563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-home\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:24.968778 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.968663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-model-cache\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:24.968778 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.968709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd2869f-9760-4b58-bee2-e6e911628300-tls-certs\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:24.968778 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.968731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-tmp-dir\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:24.968778 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.968748    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pncw9\" (UniqueName: \"kubernetes.io/projected/2cd2869f-9760-4b58-bee2-e6e911628300-kube-api-access-pncw9\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:24.968921 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:24.968854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-dshm\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.070601 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.070556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-dshm\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.070804 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.070626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-home\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.070804 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.070708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-model-cache\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.070804 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.070763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd2869f-9760-4b58-bee2-e6e911628300-tls-certs\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.070804 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.070796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-tmp-dir\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.071070 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.070826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pncw9\" (UniqueName: \"kubernetes.io/projected/2cd2869f-9760-4b58-bee2-e6e911628300-kube-api-access-pncw9\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.071208 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.071180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-home\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.071402 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.071371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-model-cache\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.071516 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.071266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-tmp-dir\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.073298 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.073268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-dshm\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.073669 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.073647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd2869f-9760-4b58-bee2-e6e911628300-tls-certs\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.082153 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.081957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pncw9\" (UniqueName: \"kubernetes.io/projected/2cd2869f-9760-4b58-bee2-e6e911628300-kube-api-access-pncw9\") pod \"ipp-route-2cf641-kserve-64694b979-q7c6c\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") " pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.160397 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.160285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:25.326516 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.326476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"]
Jul 27 17:09:25.329714 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:09:25.329681    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cd2869f_9760_4b58_bee2_e6e911628300.slice/crio-f2d3acc327ddfe32600451259e93393c6ef004b9858384b42668baa36c28405e WatchSource:0}: Error finding container f2d3acc327ddfe32600451259e93393c6ef004b9858384b42668baa36c28405e: Status 404 returned error can't find the container with id f2d3acc327ddfe32600451259e93393c6ef004b9858384b42668baa36c28405e
Jul 27 17:09:25.908340 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.908295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c" event={"ID":"2cd2869f-9760-4b58-bee2-e6e911628300","Type":"ContainerStarted","Data":"bcf9c4393fd0649730807726e2f3dc9853fce1fd8e1175d31e95e12426ef7d1b"}
Jul 27 17:09:25.908340 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.908346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c" event={"ID":"2cd2869f-9760-4b58-bee2-e6e911628300","Type":"ContainerStarted","Data":"f2d3acc327ddfe32600451259e93393c6ef004b9858384b42668baa36c28405e"}
Jul 27 17:09:25.928340 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:25.928288    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c" podStartSLOduration=1.9282671279999999 podStartE2EDuration="1.928267128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:25.926223134 +0000 UTC m=+3143.551404088" watchObservedRunningTime="2026-07-27 17:09:25.928267128 +0000 UTC m=+3143.553448148"
Jul 27 17:09:28.667121 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:28.667084    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:09:35.161288 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:35.161247    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:35.161288 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:35.161301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:35.174376 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:35.174342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:35.970700 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:35.970667    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:09:55.072371 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:09:55.072339    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:10:07.484066 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.484028    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"]
Jul 27 17:10:07.488571 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.488532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.491509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.491480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-7787cf-openshift-default-dockercfg-whcsr\""
Jul 27 17:10:07.498479 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.498440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"]
Jul 27 17:10:07.584427 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-socket\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qqz\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-kube-api-access-c9qqz\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-credential-socket\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584645 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-data\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584881 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-podinfo\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584881 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-token\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584881 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-envoy\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.584881 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.584767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-certs\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.685902 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.685858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-envoy\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.685902 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.685903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-certs\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.685956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.685978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-socket\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qqz\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-kube-api-access-c9qqz\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-credential-socket\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-data\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-podinfo\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686227 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-token\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686594 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-certs\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686594 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-credential-socket\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-socket\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686702 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-data\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.686971 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.686843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.688621 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.688593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-envoy\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.689050 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.689030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-podinfo\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.696757 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.696715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-token\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.696906 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.696851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qqz\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-kube-api-access-c9qqz\") pod \"e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") " pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.801445 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.801337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:07.953916 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.953850    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"]
Jul 27 17:10:07.956662 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:10:07.956629    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f6c68c_9bdd_4890_9a39_1b3ad04432b3.slice/crio-07171ba53534efcd0824fdd6c3d6f39627e8828908447ceac8564e771f15364c WatchSource:0}: Error finding container 07171ba53534efcd0824fdd6c3d6f39627e8828908447ceac8564e771f15364c: Status 404 returned error can't find the container with id 07171ba53534efcd0824fdd6c3d6f39627e8828908447ceac8564e771f15364c
Jul 27 17:10:07.959533 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.959493    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:10:07.959664 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.959582    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:10:07.959664 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:07.959626    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:10:08.121049 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:08.120905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" event={"ID":"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3","Type":"ContainerStarted","Data":"10bb06aed88f863096e20d723eb786a9e47f7d23eaa08f63163337fdddbb0778"}
Jul 27 17:10:08.121049 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:08.120974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" event={"ID":"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3","Type":"ContainerStarted","Data":"07171ba53534efcd0824fdd6c3d6f39627e8828908447ceac8564e771f15364c"}
Jul 27 17:10:08.143720 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:08.143654    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" podStartSLOduration=1.143633081 podStartE2EDuration="1.143633081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:08.140681834 +0000 UTC m=+3185.765862804" watchObservedRunningTime="2026-07-27 17:10:08.143633081 +0000 UTC m=+3185.768814033"
Jul 27 17:10:08.801821 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:08.801782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:09.807390 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:09.807357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:10.131852 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:10.131763    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:10.132940 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:10.132914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:20.873263 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:20.873214    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:10:30.234934 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:30.234898    2579 generic.go:356] "Generic (PLEG): container finished" podID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerID="10bb06aed88f863096e20d723eb786a9e47f7d23eaa08f63163337fdddbb0778" exitCode=0
Jul 27 17:10:30.235396 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:30.234978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" event={"ID":"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3","Type":"ContainerDied","Data":"10bb06aed88f863096e20d723eb786a9e47f7d23eaa08f63163337fdddbb0778"}
Jul 27 17:10:30.235449 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:30.235433    2579 scope.go:117] "RemoveContainer" containerID="10bb06aed88f863096e20d723eb786a9e47f7d23eaa08f63163337fdddbb0778"
Jul 27 17:10:30.236659 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:30.236625    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:10:30.236716 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:30.236703    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:10:30.236756 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:30.236739    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 17:10:31.242109 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:31.242072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" event={"ID":"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3","Type":"ContainerStarted","Data":"452e4d963a4da7a74e866187cc65ca46e3342a5b4454c5f34f97caeb64478439"}
Jul 27 17:10:31.802180 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:31.802138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:31.802437 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:31.802413    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 17:10:31.802535 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:31.802486    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 17:10:32.802530 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:32.802491    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 17:10:32.802940 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:32.802563    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 17:10:33.802635 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:33.802594    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 17:10:33.803197 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:33.802681    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 17:10:34.802753 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:34.802718    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Jul 27 17:10:34.803317 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:34.802776    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Jul 27 17:10:35.806598 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:35.806565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:35.806985 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:35.806857    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:35.807720 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:35.807702    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:48.075424 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:48.075381    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:10:53.131916 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:53.131769    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"]
Jul 27 17:10:53.132379 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:53.132038    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy" containerID="cri-o://452e4d963a4da7a74e866187cc65ca46e3342a5b4454c5f34f97caeb64478439" gracePeriod=30
Jul 27 17:10:58.389018 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.388962    2579 generic.go:356] "Generic (PLEG): container finished" podID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerID="452e4d963a4da7a74e866187cc65ca46e3342a5b4454c5f34f97caeb64478439" exitCode=0
Jul 27 17:10:58.389406 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.389044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" event={"ID":"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3","Type":"ContainerDied","Data":"452e4d963a4da7a74e866187cc65ca46e3342a5b4454c5f34f97caeb64478439"}
Jul 27 17:10:58.389406 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.389124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s" event={"ID":"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3","Type":"ContainerDied","Data":"07171ba53534efcd0824fdd6c3d6f39627e8828908447ceac8564e771f15364c"}
Jul 27 17:10:58.389406 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.389145    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07171ba53534efcd0824fdd6c3d6f39627e8828908447ceac8564e771f15364c"
Jul 27 17:10:58.389406 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.389162    2579 scope.go:117] "RemoveContainer" containerID="10bb06aed88f863096e20d723eb786a9e47f7d23eaa08f63163337fdddbb0778"
Jul 27 17:10:58.391570 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.391547    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:58.516617 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istiod-ca-cert\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.516788 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-envoy\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.516788 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516659    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-certs\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.516788 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-podinfo\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.516788 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-data\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.516788 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-socket\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.517114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516800    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-token\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.517114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9qqz\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-kube-api-access-c9qqz\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.517114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.516963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-credential-socket\") pod \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\" (UID: \"b2f6c68c-9bdd-4890-9a39-1b3ad04432b3\") "
Jul 27 17:10:58.517114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517067    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:10:58.517114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:10:58.517448 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517355    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istiod-ca-cert\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.517448 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:10:58.517448 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517374    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.517890 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:10:58.517890 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.517838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:10:58.519257 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.519220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:10:58.519474 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.519455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-kube-api-access-c9qqz" (OuterVolumeSpecName: "kube-api-access-c9qqz") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "kube-api-access-c9qqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:10:58.519797 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.519756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:10:58.519797 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.519751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" (UID: "b2f6c68c-9bdd-4890-9a39-1b3ad04432b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:10:58.619053 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.618984    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-data\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.619053 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.619050    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-workload-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.619053 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.619063    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.619310 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.619072    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9qqz\" (UniqueName: \"kubernetes.io/projected/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-kube-api-access-c9qqz\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.619310 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.619082    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-credential-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.619310 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.619095    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-envoy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:58.619310 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:58.619106    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3-istio-podinfo\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:10:59.396204 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:59.396117    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"
Jul 27 17:10:59.449251 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:59.449213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"]
Jul 27 17:10:59.460482 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:10:59.460445    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7787cf-openshift-default-84bf9746c6-dxm6s"]
Jul 27 17:11:00.968087 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:00.968050    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" path="/var/lib/kubelet/pods/b2f6c68c-9bdd-4890-9a39-1b3ad04432b3/volumes"
Jul 27 17:11:11.966420 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:11.966384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:11:22.309208 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:22.309166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"]
Jul 27 17:11:25.965755 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:25.965712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:11:37.330062 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.329980    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c" podUID="2cd2869f-9760-4b58-bee2-e6e911628300" containerName="main" containerID="cri-o://bcf9c4393fd0649730807726e2f3dc9853fce1fd8e1175d31e95e12426ef7d1b" gracePeriod=45
Jul 27 17:11:37.589193 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.589101    2579 generic.go:356] "Generic (PLEG): container finished" podID="2cd2869f-9760-4b58-bee2-e6e911628300" containerID="bcf9c4393fd0649730807726e2f3dc9853fce1fd8e1175d31e95e12426ef7d1b" exitCode=0
Jul 27 17:11:37.589193 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.589165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c" event={"ID":"2cd2869f-9760-4b58-bee2-e6e911628300","Type":"ContainerDied","Data":"bcf9c4393fd0649730807726e2f3dc9853fce1fd8e1175d31e95e12426ef7d1b"}
Jul 27 17:11:37.589378 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.589206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c" event={"ID":"2cd2869f-9760-4b58-bee2-e6e911628300","Type":"ContainerDied","Data":"f2d3acc327ddfe32600451259e93393c6ef004b9858384b42668baa36c28405e"}
Jul 27 17:11:37.589378 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.589219    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2d3acc327ddfe32600451259e93393c6ef004b9858384b42668baa36c28405e"
Jul 27 17:11:37.597153 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.597121    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:11:37.707666 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707625    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-model-cache\") pod \"2cd2869f-9760-4b58-bee2-e6e911628300\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") "
Jul 27 17:11:37.707850 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-tmp-dir\") pod \"2cd2869f-9760-4b58-bee2-e6e911628300\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") "
Jul 27 17:11:37.707850 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-dshm\") pod \"2cd2869f-9760-4b58-bee2-e6e911628300\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") "
Jul 27 17:11:37.707850 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-home\") pod \"2cd2869f-9760-4b58-bee2-e6e911628300\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") "
Jul 27 17:11:37.707989 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd2869f-9760-4b58-bee2-e6e911628300-tls-certs\") pod \"2cd2869f-9760-4b58-bee2-e6e911628300\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") "
Jul 27 17:11:37.707989 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pncw9\" (UniqueName: \"kubernetes.io/projected/2cd2869f-9760-4b58-bee2-e6e911628300-kube-api-access-pncw9\") pod \"2cd2869f-9760-4b58-bee2-e6e911628300\" (UID: \"2cd2869f-9760-4b58-bee2-e6e911628300\") "
Jul 27 17:11:37.707989 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.707917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cd2869f-9760-4b58-bee2-e6e911628300" (UID: "2cd2869f-9760-4b58-bee2-e6e911628300"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:11:37.708147 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.708058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cd2869f-9760-4b58-bee2-e6e911628300" (UID: "2cd2869f-9760-4b58-bee2-e6e911628300"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:11:37.708147 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.708134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-home" (OuterVolumeSpecName: "home") pod "2cd2869f-9760-4b58-bee2-e6e911628300" (UID: "2cd2869f-9760-4b58-bee2-e6e911628300"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:11:37.708250 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.708235    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-model-cache\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:11:37.708291 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.708253    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-tmp-dir\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:11:37.708291 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.708262    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-home\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:11:37.710329 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.710304    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-dshm" (OuterVolumeSpecName: "dshm") pod "2cd2869f-9760-4b58-bee2-e6e911628300" (UID: "2cd2869f-9760-4b58-bee2-e6e911628300"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:11:37.710433 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.710401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd2869f-9760-4b58-bee2-e6e911628300-kube-api-access-pncw9" (OuterVolumeSpecName: "kube-api-access-pncw9") pod "2cd2869f-9760-4b58-bee2-e6e911628300" (UID: "2cd2869f-9760-4b58-bee2-e6e911628300"). InnerVolumeSpecName "kube-api-access-pncw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:11:37.710482 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.710453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd2869f-9760-4b58-bee2-e6e911628300-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cd2869f-9760-4b58-bee2-e6e911628300" (UID: "2cd2869f-9760-4b58-bee2-e6e911628300"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:11:37.809549 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.809504    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cd2869f-9760-4b58-bee2-e6e911628300-tls-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:11:37.809549 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.809543    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pncw9\" (UniqueName: \"kubernetes.io/projected/2cd2869f-9760-4b58-bee2-e6e911628300-kube-api-access-pncw9\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:11:37.809549 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:37.809553    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cd2869f-9760-4b58-bee2-e6e911628300-dshm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:11:38.593728 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:38.593690    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"
Jul 27 17:11:38.619840 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:38.619798    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"]
Jul 27 17:11:38.622841 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:38.622799    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2cf641/ipp-route-2cf641-kserve-64694b979-q7c6c"]
Jul 27 17:11:38.967383 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:11:38.967337    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd2869f-9760-4b58-bee2-e6e911628300" path="/var/lib/kubelet/pods/2cd2869f-9760-4b58-bee2-e6e911628300/volumes"
Jul 27 17:12:03.039506 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:03.039473    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:12:03.046880 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:03.046837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:12:06.834298 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:06.834260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"]
Jul 27 17:12:06.834815 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:06.834489    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" containerID="cri-o://35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67" gracePeriod=30
Jul 27 17:12:08.089083 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:08.089036    2579 patch_prober.go:28] interesting pod/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 17:12:08.089551 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:08.089095    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:12:12.094793 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.094721    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:12:12.143509 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.143472    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-socket\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.143694 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.143572    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng6cb\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-kube-api-access-ng6cb\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.143694 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.143595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-envoy\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.143694 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.143624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-data\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.143694 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.143655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-token\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.143899 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.143827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:12.144149 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istiod-ca-cert\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.144276 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-certs\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.144276 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:12:12.144646 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144620    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:12.144646 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144637    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:12.144761 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-credential-socket\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.144761 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-podinfo\") pod \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\" (UID: \"8d3b6392-9e15-4b36-8ee1-b71bf781190a\") "
Jul 27 17:12:12.144883 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.144858    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:12.145374 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.145350    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-data\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.145484 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.145380    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istiod-ca-cert\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.145484 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.145395    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.145484 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.145408    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-credential-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.145484 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.145423    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-workload-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.146247 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.146221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:12.146670 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.146645    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:12:12.146804 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.146786    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-kube-api-access-ng6cb" (OuterVolumeSpecName: "kube-api-access-ng6cb") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "kube-api-access-ng6cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:12:12.147155 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.147126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d3b6392-9e15-4b36-8ee1-b71bf781190a" (UID: "8d3b6392-9e15-4b36-8ee1-b71bf781190a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:12:12.245968 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.245928    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng6cb\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-kube-api-access-ng6cb\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.245968 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.245959    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-envoy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.245968 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.245968    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.245968 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.245979    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d3b6392-9e15-4b36-8ee1-b71bf781190a-istio-podinfo\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:12:12.767850 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.767813    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerID="35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67" exitCode=0
Jul 27 17:12:12.768075 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.767873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"
Jul 27 17:12:12.768075 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.767908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" event={"ID":"8d3b6392-9e15-4b36-8ee1-b71bf781190a","Type":"ContainerDied","Data":"35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67"}
Jul 27 17:12:12.768075 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.767995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz" event={"ID":"8d3b6392-9e15-4b36-8ee1-b71bf781190a","Type":"ContainerDied","Data":"608f95a4b1884ecd41193d553cb19d4ba7d9417b0fdc0e1f3676f64d61592aeb"}
Jul 27 17:12:12.768075 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.768037    2579 scope.go:117] "RemoveContainer" containerID="35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67"
Jul 27 17:12:12.777760 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.777737    2579 scope.go:117] "RemoveContainer" containerID="8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"
Jul 27 17:12:12.786883 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.786865    2579 scope.go:117] "RemoveContainer" containerID="35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67"
Jul 27 17:12:12.787238 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:12:12.787216    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67\": container with ID starting with 35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67 not found: ID does not exist" containerID="35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67"
Jul 27 17:12:12.787354 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.787251    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67"} err="failed to get container status \"35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67\": rpc error: code = NotFound desc = could not find container \"35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67\": container with ID starting with 35861952d4f3a729d4cbce3a64053f26a16f0c1d44729335d67f7daccdd7fc67 not found: ID does not exist"
Jul 27 17:12:12.787354 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.787275    2579 scope.go:117] "RemoveContainer" containerID="8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"
Jul 27 17:12:12.787540 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:12:12.787522    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451\": container with ID starting with 8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451 not found: ID does not exist" containerID="8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"
Jul 27 17:12:12.787579 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.787548    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451"} err="failed to get container status \"8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451\": rpc error: code = NotFound desc = could not find container \"8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451\": container with ID starting with 8cfc852c3a783a7887bcd274865264a28a0949e4da9f127ea018f0eed38e7451 not found: ID does not exist"
Jul 27 17:12:12.794573 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.794538    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"]
Jul 27 17:12:12.798859 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.798823    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cf641-openshift-default-578dc67d54-rtkpz"]
Jul 27 17:12:12.968131 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:12.968095    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" path="/var/lib/kubelet/pods/8d3b6392-9e15-4b36-8ee1-b71bf781190a/volumes"
Jul 27 17:12:20.852782 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:20.852740    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:12:45.059069 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:12:45.058959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:13:06.054806 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:06.054764    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-x2l82"]
Jul 27 17:13:14.082380 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:14.082340    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"]
Jul 27 17:13:14.082919 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:14.082682    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" containerID="cri-o://94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088" gracePeriod=30
Jul 27 17:13:15.797704 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:15.797666    2579 patch_prober.go:28] interesting pod/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 17:13:15.798156 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:15.797747    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:13:19.345899 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.345868    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 17:13:19.407703 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-token\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.407703 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-podinfo\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.407703 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407683    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-certs\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.407703 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407704    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009d373e-8fef-4f9a-84f3-f9c7a7180607-istiod-ca-cert\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.408085 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407826    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-data\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.408085 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407866    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-envoy\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.408085 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407904    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-socket\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.408085 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.407934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-credential-socket\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.408085 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408014    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzfpm\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-kube-api-access-nzfpm\") pod \"009d373e-8fef-4f9a-84f3-f9c7a7180607\" (UID: \"009d373e-8fef-4f9a-84f3-f9c7a7180607\") "
Jul 27 17:13:19.408085 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:13:19.408413 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009d373e-8fef-4f9a-84f3-f9c7a7180607-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:13:19.408413 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:13:19.408886 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408671    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:13:19.408886 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408757    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-certs\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.408886 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408780    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009d373e-8fef-4f9a-84f3-f9c7a7180607-istiod-ca-cert\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.408886 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.408794    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-workload-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.409201 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.409111    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-data" (OuterVolumeSpecName: "istio-data") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:13:19.410244 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.410216    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:13:19.410244 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.410230    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-token" (OuterVolumeSpecName: "istio-token") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:13:19.410381 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.410301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-kube-api-access-nzfpm" (OuterVolumeSpecName: "kube-api-access-nzfpm") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "kube-api-access-nzfpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:13:19.410459 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.410439    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "009d373e-8fef-4f9a-84f3-f9c7a7180607" (UID: "009d373e-8fef-4f9a-84f3-f9c7a7180607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:13:19.510101 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.510043    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzfpm\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-kube-api-access-nzfpm\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.510101 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.510098    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-token\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.510101 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.510117    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-podinfo\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.510366 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.510129    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-data\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.510366 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.510144    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-istio-envoy\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:19.510366 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:19.510153    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009d373e-8fef-4f9a-84f3-f9c7a7180607-credential-socket\") on node \"ip-10-0-129-154.ec2.internal\" DevicePath \"\""
Jul 27 17:13:20.104920 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.104876    2579 generic.go:356] "Generic (PLEG): container finished" podID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerID="94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088" exitCode=0
Jul 27 17:13:20.105134 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.104989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"
Jul 27 17:13:20.105134 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.105038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" event={"ID":"009d373e-8fef-4f9a-84f3-f9c7a7180607","Type":"ContainerDied","Data":"94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088"}
Jul 27 17:13:20.105134 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.105102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt" event={"ID":"009d373e-8fef-4f9a-84f3-f9c7a7180607","Type":"ContainerDied","Data":"592c03a25cb00178c5dc3896bb2d7985984888ec75fa9e5f1db4e283d88f2538"}
Jul 27 17:13:20.105134 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.105126    2579 scope.go:117] "RemoveContainer" containerID="94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088"
Jul 27 17:13:20.115261 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.115234    2579 scope.go:117] "RemoveContainer" containerID="0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"
Jul 27 17:13:20.124365 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.124344    2579 scope.go:117] "RemoveContainer" containerID="94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088"
Jul 27 17:13:20.124716 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:13:20.124689    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088\": container with ID starting with 94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088 not found: ID does not exist" containerID="94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088"
Jul 27 17:13:20.124812 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.124729    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088"} err="failed to get container status \"94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088\": rpc error: code = NotFound desc = could not find container \"94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088\": container with ID starting with 94f0dc367be7f2359391e3120c4a6f7a63e5b629f2e5f7781b194f0d1d48a088 not found: ID does not exist"
Jul 27 17:13:20.124812 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.124756    2579 scope.go:117] "RemoveContainer" containerID="0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"
Jul 27 17:13:20.125179 ip-10-0-129-154 kubenswrapper[2579]: E0727 17:13:20.125136    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae\": container with ID starting with 0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae not found: ID does not exist" containerID="0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"
Jul 27 17:13:20.125308 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.125177    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae"} err="failed to get container status \"0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae\": rpc error: code = NotFound desc = could not find container \"0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae\": container with ID starting with 0233b14fd18bf750895fe7306f2aa520607159d28b8e7e84fc1ac94a1d88afae not found: ID does not exist"
Jul 27 17:13:20.128562 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.128536    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"]
Jul 27 17:13:20.130252 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.130231    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d4529a-openshift-default-9bb75cbb4-pdnkt"]
Jul 27 17:13:20.967949 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:20.967908    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" path="/var/lib/kubelet/pods/009d373e-8fef-4f9a-84f3-f9c7a7180607/volumes"
Jul 27 17:13:21.000544 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:21.000509    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-bz8jd_09b45e37-3846-4ea5-be60-cdf5b6e0f458/copy-manifests/0.log"
Jul 27 17:13:21.008606 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:21.008580    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-bz8jd_09b45e37-3846-4ea5-be60-cdf5b6e0f458/manager/0.log"
Jul 27 17:13:23.952834 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:23.952802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-x2l82_5ae6a93c-61e4-4d8b-a493-de25d5d31240/limitador/0.log"
Jul 27 17:13:24.664077 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:24.664047    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-4cj5x_9b4199b8-d637-495d-95ba-a680a2e4c59c/kube-auth-proxy/0.log"
Jul 27 17:13:24.890717 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:24.890683    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/1.log"
Jul 27 17:13:25.260395 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:25.260356    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8669d845dc-k4ppl_cbe76343-1d2e-4430-adc8-c87ac6c49546/router/0.log"
Jul 27 17:13:26.114847 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:26.114814    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-f8w9b_f3427f5f-fba3-415a-bf6a-c3f0637ffa38/main/0.log"
Jul 27 17:13:26.231547 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:26.231521    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-v9qgp_5b0fe6ff-a7ff-452d-91f9-9861471e2d99/main/0.log"
Jul 27 17:13:31.760803 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.760765    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ddw25/must-gather-5ktq6"]
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761147    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd2869f-9760-4b58-bee2-e6e911628300" containerName="main"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761159    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd2869f-9760-4b58-bee2-e6e911628300" containerName="main"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761173    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761178    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761193    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761199    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761205    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761210    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761216    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761223    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761228    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761234    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy"
Jul 27 17:13:31.761233 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761240    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761245    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761300    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761306    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd2869f-9760-4b58-bee2-e6e911628300" containerName="main"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761313    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761320    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d3b6392-9e15-4b36-8ee1-b71bf781190a" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761325    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761330    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="009d373e-8fef-4f9a-84f3-f9c7a7180607" containerName="istio-proxy"
Jul 27 17:13:31.761627 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.761336    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f6c68c-9bdd-4890-9a39-1b3ad04432b3" containerName="istio-proxy"
Jul 27 17:13:31.765900 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.765877    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:31.769020 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.768974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ddw25\"/\"default-dockercfg-nqbtv\""
Jul 27 17:13:31.769173 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.769047    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddw25\"/\"openshift-service-ca.crt\""
Jul 27 17:13:31.769801 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.769782    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ddw25\"/\"kube-root-ca.crt\""
Jul 27 17:13:31.775924 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.775899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/must-gather-5ktq6"]
Jul 27 17:13:31.829855 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.829824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f5867766-3925-4344-b124-9cc3edfc6dc4-must-gather-output\") pod \"must-gather-5ktq6\" (UID: \"f5867766-3925-4344-b124-9cc3edfc6dc4\") " pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:31.830098 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.829888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttd4\" (UniqueName: \"kubernetes.io/projected/f5867766-3925-4344-b124-9cc3edfc6dc4-kube-api-access-4ttd4\") pod \"must-gather-5ktq6\" (UID: \"f5867766-3925-4344-b124-9cc3edfc6dc4\") " pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:31.930682 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.930643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f5867766-3925-4344-b124-9cc3edfc6dc4-must-gather-output\") pod \"must-gather-5ktq6\" (UID: \"f5867766-3925-4344-b124-9cc3edfc6dc4\") " pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:31.930853 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.930719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttd4\" (UniqueName: \"kubernetes.io/projected/f5867766-3925-4344-b124-9cc3edfc6dc4-kube-api-access-4ttd4\") pod \"must-gather-5ktq6\" (UID: \"f5867766-3925-4344-b124-9cc3edfc6dc4\") " pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:31.931050 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.930994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f5867766-3925-4344-b124-9cc3edfc6dc4-must-gather-output\") pod \"must-gather-5ktq6\" (UID: \"f5867766-3925-4344-b124-9cc3edfc6dc4\") " pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:31.950527 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:31.950491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttd4\" (UniqueName: \"kubernetes.io/projected/f5867766-3925-4344-b124-9cc3edfc6dc4-kube-api-access-4ttd4\") pod \"must-gather-5ktq6\" (UID: \"f5867766-3925-4344-b124-9cc3edfc6dc4\") " pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:32.076171 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:32.076070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/must-gather-5ktq6"
Jul 27 17:13:32.248216 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:32.248179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/must-gather-5ktq6"]
Jul 27 17:13:32.252194 ip-10-0-129-154 kubenswrapper[2579]: W0727 17:13:32.252158    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5867766_3925_4344_b124_9cc3edfc6dc4.slice/crio-4ef3f02833974dc3bd5b1777535e589699ba592bcd8bf58917d4f964db7363d3 WatchSource:0}: Error finding container 4ef3f02833974dc3bd5b1777535e589699ba592bcd8bf58917d4f964db7363d3: Status 404 returned error can't find the container with id 4ef3f02833974dc3bd5b1777535e589699ba592bcd8bf58917d4f964db7363d3
Jul 27 17:13:32.254245 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:32.254227    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:13:33.175559 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:33.175518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/must-gather-5ktq6" event={"ID":"f5867766-3925-4344-b124-9cc3edfc6dc4","Type":"ContainerStarted","Data":"4ef3f02833974dc3bd5b1777535e589699ba592bcd8bf58917d4f964db7363d3"}
Jul 27 17:13:34.187240 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:34.187196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/must-gather-5ktq6" event={"ID":"f5867766-3925-4344-b124-9cc3edfc6dc4","Type":"ContainerStarted","Data":"f2cc250141c82326ae7584675b00650cd961f5436609bcf0c3e451400197fc1e"}
Jul 27 17:13:34.187684 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:34.187256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/must-gather-5ktq6" event={"ID":"f5867766-3925-4344-b124-9cc3edfc6dc4","Type":"ContainerStarted","Data":"c746040741dac7fb0d63c1d21e12a46e83d5d27a3e1d4ed0454b4c545fac686e"}
Jul 27 17:13:34.222099 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:34.222020    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ddw25/must-gather-5ktq6" podStartSLOduration=2.384075495 podStartE2EDuration="3.221971635s" totalImagesPullingTime="837.89614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:13:34.217971883 +0000 UTC m=+3391.843152836" watchObservedRunningTime="2026-07-27 17:13:34.221971635 +0000 UTC m=+3391.847152591"
Jul 27 17:13:34.923673 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:34.923633    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2vv29_09e88a58-768b-4d90-80f6-1cfc0a304f65/global-pull-secret-syncer/0.log"
Jul 27 17:13:35.163072 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:35.163037    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r5nnt_efeb8dbd-2c92-40d1-8ac2-cb53f9756db4/konnectivity-agent/0.log"
Jul 27 17:13:35.273635 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:35.273590    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-154.ec2.internal_b14883ed3c5020af2ec5d61c737e792e/haproxy/0.log"
Jul 27 17:13:41.375472 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:41.375407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-x2l82_5ae6a93c-61e4-4d8b-a493-de25d5d31240/limitador/0.log"
Jul 27 17:13:43.564475 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:43.564364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-ck5sc_65b258f3-467c-4a00-9f74-7eda7c895684/kube-state-metrics/0.log"
Jul 27 17:13:43.603626 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:43.603593    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-ck5sc_65b258f3-467c-4a00-9f74-7eda7c895684/kube-rbac-proxy-main/0.log"
Jul 27 17:13:43.638261 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:43.638224    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-ck5sc_65b258f3-467c-4a00-9f74-7eda7c895684/kube-rbac-proxy-self/0.log"
Jul 27 17:13:43.750967 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:43.750937    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hslt_4234a8fe-a8f2-4e81-8569-960422de412e/node-exporter/0.log"
Jul 27 17:13:43.791625 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:43.791589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hslt_4234a8fe-a8f2-4e81-8569-960422de412e/kube-rbac-proxy/0.log"
Jul 27 17:13:43.831458 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:43.831372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hslt_4234a8fe-a8f2-4e81-8569-960422de412e/init-textfile/0.log"
Jul 27 17:13:44.082895 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.082817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-98wv5_4749b905-2dfa-4408-b1a9-d66436bf5b25/kube-rbac-proxy-main/0.log"
Jul 27 17:13:44.125143 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.125118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-98wv5_4749b905-2dfa-4408-b1a9-d66436bf5b25/kube-rbac-proxy-self/0.log"
Jul 27 17:13:44.152534 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.152504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-98wv5_4749b905-2dfa-4408-b1a9-d66436bf5b25/openshift-state-metrics/0.log"
Jul 27 17:13:44.200230 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.200198    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/prometheus/0.log"
Jul 27 17:13:44.228153 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.228119    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/config-reloader/0.log"
Jul 27 17:13:44.255110 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.255063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/thanos-sidecar/0.log"
Jul 27 17:13:44.289176 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.289147    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/kube-rbac-proxy-web/0.log"
Jul 27 17:13:44.314202 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.314173    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/kube-rbac-proxy/0.log"
Jul 27 17:13:44.344792 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.344698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/kube-rbac-proxy-thanos/0.log"
Jul 27 17:13:44.369609 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.369576    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5b37404e-bec6-44eb-979d-58213529b64d/init-config-reloader/0.log"
Jul 27 17:13:44.484479 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.484449    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55bdd94856-rvs42_f71e5436-f3aa-4a52-ac4d-c91319a29420/telemeter-client/0.log"
Jul 27 17:13:44.508670 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.508641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55bdd94856-rvs42_f71e5436-f3aa-4a52-ac4d-c91319a29420/reload/0.log"
Jul 27 17:13:44.537900 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:44.537875    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55bdd94856-rvs42_f71e5436-f3aa-4a52-ac4d-c91319a29420/kube-rbac-proxy/0.log"
Jul 27 17:13:45.251566 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.251529    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"]
Jul 27 17:13:45.260344 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.260308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.269410 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.269365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"]
Jul 27 17:13:45.293114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.292703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-lib-modules\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.293114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.292755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-proc\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.293114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.292798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-podres\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.293114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.292824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-sys\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.293114 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.292864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmtk\" (UniqueName: \"kubernetes.io/projected/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-kube-api-access-nsmtk\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394108 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-lib-modules\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394400 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-proc\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394574 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-lib-modules\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394574 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-proc\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394707 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-podres\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394707 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-sys\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394823 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-sys\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394913 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-podres\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.394988 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.394678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmtk\" (UniqueName: \"kubernetes.io/projected/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-kube-api-access-nsmtk\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.407398 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.407358    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmtk\" (UniqueName: \"kubernetes.io/projected/c5dee49c-ebeb-4cac-9df4-37fb8f4754c9-kube-api-access-nsmtk\") pod \"perf-node-gather-daemonset-qsvg9\" (UID: \"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9\") " pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.575904 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.575804    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:45.778213 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:45.778172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"]
Jul 27 17:13:46.303830 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:46.303793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9" event={"ID":"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9","Type":"ContainerStarted","Data":"9719fbe493e0f440aa794eede981699b7413a0b48216c51537c7e1135494ff61"}
Jul 27 17:13:46.304328 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:46.303851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9" event={"ID":"c5dee49c-ebeb-4cac-9df4-37fb8f4754c9","Type":"ContainerStarted","Data":"f28c53e73d698810d68177a18317c9e5a24b7c52b28d3f10b5c80f9104418e33"}
Jul 27 17:13:46.304328 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:46.303901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:46.323775 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:46.323721    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9" podStartSLOduration=1.32370357 podStartE2EDuration="1.32370357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:13:46.321413815 +0000 UTC m=+3403.946594793" watchObservedRunningTime="2026-07-27 17:13:46.32370357 +0000 UTC m=+3403.948884576"
Jul 27 17:13:46.647314 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:46.647190    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/2.log"
Jul 27 17:13:46.652333 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:46.652306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-tw85l_4eeec8f8-f1c6-4692-acab-1dfa7aca2143/console-operator/3.log"
Jul 27 17:13:47.996225 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:47.996195    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-qmfvb_23882235-2eb7-4b36-a10c-107bd62ac2e3/volume-data-source-validator/0.log"
Jul 27 17:13:49.312219 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:49.312183    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pr7qc_799b4fdb-c34f-434f-977d-323105f3709b/dns/0.log"
Jul 27 17:13:49.357380 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:49.357279    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pr7qc_799b4fdb-c34f-434f-977d-323105f3709b/kube-rbac-proxy/0.log"
Jul 27 17:13:49.416944 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:49.416916    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klbl8_a09ef6bc-7e7b-4803-bb17-325310a8cf7f/dns-node-resolver/0.log"
Jul 27 17:13:50.103316 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:50.103280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d9fc4f64d-fxjtl_9a107051-009c-43cb-84f3-26a86f62a3e9/registry/0.log"
Jul 27 17:13:50.143719 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:50.143688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-btrf8_dde7bb49-725e-45b3-b4bb-53fff8ad3516/node-ca/0.log"
Jul 27 17:13:51.717142 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:51.717112    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-4cj5x_9b4199b8-d637-495d-95ba-a680a2e4c59c/kube-auth-proxy/0.log"
Jul 27 17:13:51.836873 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:51.836837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/0.log"
Jul 27 17:13:51.869099 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:51.869059    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/1.log"
Jul 27 17:13:52.021994 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:52.021909    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8669d845dc-k4ppl_cbe76343-1d2e-4430-adc8-c87ac6c49546/router/0.log"
Jul 27 17:13:52.326774 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:52.326690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ddw25/perf-node-gather-daemonset-qsvg9"
Jul 27 17:13:52.941021 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:52.940870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tx9vb_5b4bb02e-b6a1-433f-9a49-f1e5882aa4e7/serve-healthcheck-canary/0.log"
Jul 27 17:13:53.499447 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:53.499411    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sqz4_f50b064e-fb6d-4492-8a3d-d873d8468524/kube-rbac-proxy/0.log"
Jul 27 17:13:53.521638 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:53.521601    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sqz4_f50b064e-fb6d-4492-8a3d-d873d8468524/exporter/0.log"
Jul 27 17:13:53.544090 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:53.544057    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5sqz4_f50b064e-fb6d-4492-8a3d-d873d8468524/extractor/0.log"
Jul 27 17:13:56.819771 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:56.819729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-4cj5x_9b4199b8-d637-495d-95ba-a680a2e4c59c/kube-auth-proxy/0.log"
Jul 27 17:13:56.870063 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:56.870030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/0.log"
Jul 27 17:13:56.907591 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:56.907505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/1.log"
Jul 27 17:13:56.998739 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:56.998688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8669d845dc-k4ppl_cbe76343-1d2e-4430-adc8-c87ac6c49546/router/0.log"
Jul 27 17:13:57.210315 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:57.210283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-bz8jd_09b45e37-3846-4ea5-be60-cdf5b6e0f458/manager/0.log"
Jul 27 17:13:57.238315 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:57.238285    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-55548c94b5-bz8jd_09b45e37-3846-4ea5-be60-cdf5b6e0f458/copy-manifests/0.log"
Jul 27 17:13:58.676852 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:58.676822    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78dbfdf94f-4cj5x_9b4199b8-d637-495d-95ba-a680a2e4c59c/kube-auth-proxy/0.log"
Jul 27 17:13:58.725407 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:58.725372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/0.log"
Jul 27 17:13:58.758062 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:58.758034    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qkq6j_1ecfd50d-aadb-4df8-9249-25f31fe1cda9/istio-proxy/1.log"
Jul 27 17:13:58.794922 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:58.794891    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-cd69fdcb8-g8c66_a6c8b3e9-05f8-41ae-8de7-4baf56d10838/manager/0.log"
Jul 27 17:13:58.868844 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:58.868801    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8669d845dc-k4ppl_cbe76343-1d2e-4430-adc8-c87ac6c49546/router/0.log"
Jul 27 17:13:58.886474 ip-10-0-129-154 kubenswrapper[2579]: I0727 17:13:58.886430    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jjqtr_a07be0e9-9d31-46d2-a5e6-9babf4cac035/openshift-lws-operator/0.log"