Sep 16 13:33:09.423406 ip-10-0-133-251 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 13:33:09.843617 ip-10-0-133-251 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:33:09.843617 ip-10-0-133-251 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 13:33:09.843617 ip-10-0-133-251 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:33:09.843617 ip-10-0-133-251 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 13:33:09.843617 ip-10-0-133-251 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 13:33:09.846607 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.846510    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 13:33:09.851752 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851729    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:33:09.851752 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851748    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:33:09.851752 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851753    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:33:09.851752 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851758    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851763    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851767    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851771    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851775    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851779    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851783    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851786    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851790    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851794    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851800    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851807    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851811    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851814    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851818    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851821    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851826    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851830    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851834    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:33:09.852003 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851838    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851842    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851847    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851853    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851859    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851866    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851871    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851883    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851888    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851893    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851898    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851902    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851906    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851911    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851916    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851923    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851928    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851932    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851937    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:33:09.852808 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851941    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851945    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851949    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851953    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851958    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851962    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851966    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851971    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851975    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851979    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851983    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851987    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851991    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851995    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.851999    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852003    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852007    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852011    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852015    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852020    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:33:09.853680 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852024    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852028    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852033    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852037    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852041    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852046    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852050    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852055    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852061    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852065    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852069    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852074    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852078    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852082    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852103    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852110    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852117    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852121    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852125    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852129    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:33:09.854537 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852133    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852137    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852142    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852145    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852149    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852154    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852158    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852162    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852166    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852170    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852174    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852178    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852183    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852187    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852191    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852195    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852199    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.852204    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853481    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:33:09.855388 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853494    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853499    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853505    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853510    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853515    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853521    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853528    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853532    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853537    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853541    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853546    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853551    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853555    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853560    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853564    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853568    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853573    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853577    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853584    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:33:09.856223 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853590    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853595    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853600    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853606    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853610    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853615    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853619    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853624    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853628    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853632    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853637    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853642    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853646    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853651    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853657    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853661    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853666    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853671    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853675    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853679    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:33:09.856903 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853684    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853688    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853692    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853696    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853700    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853704    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853708    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853713    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853717    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853721    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853725    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853729    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853734    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853738    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853742    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853746    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853750    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853754    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853758    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853763    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:33:09.857491 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853767    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853771    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853775    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853780    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853784    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853787    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853792    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853797    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853802    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853807    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853813    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853817    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853821    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853825    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853829    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853833    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853837    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853841    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853845    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853849    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:33:09.858020 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853853    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853858    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853862    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853866    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853870    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853874    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853878    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853882    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853886    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853890    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853895    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853899    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853903    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853907    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853911    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853915    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853919    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853925    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.853932    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854562    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:33:09.858557 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854572    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854578    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854583    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854587    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854592    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854596    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854600    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854604    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854608    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854612    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854617    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854621    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854625    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854629    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854634    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854638    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854642    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854646    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854651    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854655    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:33:09.859169 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854659    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854664    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854671    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854675    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854679    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854684    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854688    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854692    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854696    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854701    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854705    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854709    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854713    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854718    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854722    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854727    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854731    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854735    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854740    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854744    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:33:09.859770 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854748    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854752    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854756    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854760    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854764    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854771    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854777    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854782    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854786    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854790    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854795    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854799    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854803    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854807    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854813    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854817    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854821    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854827    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854831    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854835    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:33:09.860296 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854840    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854844    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854848    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854852    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854856    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854861    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854865    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854870    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854874    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854878    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854882    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854887    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854891    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854895    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854899    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854903    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854907    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854912    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854916    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:33:09.860893 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854920    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854925    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854929    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854933    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854937    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854944    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854950    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854957    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854962    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854966    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854973    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854977    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854982    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854987    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854991    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.854997    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.855003    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.855008    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.855012    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855140    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 13:33:09.861392 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855189    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855202    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855209    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855216    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855221    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855236    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855244    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855249    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855254    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855260    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855265    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855270    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855275    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855280    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855285    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855290    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855295    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855302    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855307    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855314    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855320    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855325    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855333    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 13:33:09.861900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855338    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855343    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855349    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855354    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855358    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855363    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855368    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855373    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855380    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855384    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855390    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855395    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855399    2556 flags.go:64] FLAG: --enable-server="true"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855404    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855411    2556 flags.go:64] FLAG: --event-burst="100"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855415    2556 flags.go:64] FLAG: --event-qps="50"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855420    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855425    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855430    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855436    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855441    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855445    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855451    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855455    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855460    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 13:33:09.862483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855464    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855469    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855473    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855479    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855483    2556 flags.go:64] FLAG: --feature-gates=""
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855489    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855494    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855499    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855504    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855509    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855514    2556 flags.go:64] FLAG: --help="false"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855519    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-133-251.ec2.internal"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855524    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855528    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855533    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855539    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855544    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855548    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855553    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855558    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855563    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855567    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855572    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855577    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 16 13:33:09.863246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855581    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855586    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855591    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855596    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855600    2556 flags.go:64] FLAG: --lock-file=""
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855605    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855610    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855615    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855624    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855628    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855633    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855638    2556 flags.go:64] FLAG: --logging-format="text"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855643    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855648    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855653    2556 flags.go:64] FLAG: --manifest-url=""
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855658    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855665    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855670    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855677    2556 flags.go:64] FLAG: --max-pods="110"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855682    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855687    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855692    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855697    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855702    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855707    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 13:33:09.863899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855711    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855723    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855728    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855734    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855739    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855744    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855752    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855757    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855762    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855767    2556 flags.go:64] FLAG: --port="10250"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855772    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855777    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b29b8126a5cbbc18"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855782    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855786    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855791    2556 flags.go:64] FLAG: --register-node="true"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855796    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855802    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855807    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855812    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855817    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855823    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855828    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855833    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855837    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855842    2556 flags.go:64] FLAG: --runonce="false"
Sep 16 13:33:09.864553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855847    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855852    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855857    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855861    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855866    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855871    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855885    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855891    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855896    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855900    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855905    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855910    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855915    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855920    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855924    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855933    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855938    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855943    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855951    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855955    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855959    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855964    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855968    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855973    2556 flags.go:64] FLAG: --v="2"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855979    2556 flags.go:64] FLAG: --version="false"
Sep 16 13:33:09.865196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855986    2556 flags.go:64] FLAG: --vmodule=""
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855992    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.855997    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856206    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856214    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856218    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856222    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856229    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856234    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856239    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856243    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856247    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856251    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856256    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856260    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856264    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856268    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856272    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856277    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:33:09.865805 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856282    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856287    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856291    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856295    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856299    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856303    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856311    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856315    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856319    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856323    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856328    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856332    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856336    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856341    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856346    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856350    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856355    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856361    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856365    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856369    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856373    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:33:09.866463 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856378    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856382    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856387    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856391    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856395    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856399    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856404    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856408    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856412    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856416    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856420    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856424    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856429    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856434    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856438    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856442    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856446    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856452    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856456    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:33:09.867206 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856460    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856464    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856468    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856472    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856476    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856480    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856484    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856489    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856492    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856498    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856502    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856506    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856511    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856515    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856519    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856525    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856531    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856536    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856540    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:33:09.867720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856544    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856548    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856553    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856557    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856561    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856565    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856569    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856574    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856578    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856583    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856587    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856593    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856597    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856603    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856609    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856614    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856617    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856622    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856626    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856630    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:33:09.868237 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856634    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856639    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856645    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.856649    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.857404    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.865358    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.865373    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.865393    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.865402    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865459    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865464    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865468    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865471    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865474    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:33:09.868726 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865477    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865480    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865483    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865485    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865488    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865492    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865494    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865497    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865499    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865502    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865504    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865507    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865509    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865512    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865514    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865517    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865519    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865522    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865525    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865527    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:33:09.869077 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865530    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865533    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865536    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865538    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865542    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865545    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865547    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865550    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865552    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865555    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865558    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865560    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865562    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865565    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865567    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865570    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865572    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865575    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865578    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865580    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:33:09.869591 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865583    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865585    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865588    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865590    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865592    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865595    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865597    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865600    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865602    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865604    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865607    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865609    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865612    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865615    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865617    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865620    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865623    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865626    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865630    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:33:09.870114 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865634    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865636    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865639    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865642    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865645    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865647    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865650    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865652    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865655    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865658    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865661    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865663    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865667    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865671    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865674    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865676    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865679    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865681    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865684    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:33:09.870583 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865686    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865689    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865691    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865695    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865699    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865703    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865705    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865708    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865711    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865714    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865717    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865720    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865723    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865726    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865729    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:33:09.871076 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.865732    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.865737    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867699    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867717    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867722    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867725    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867728    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867731    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867735    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867738    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867741    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867744    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867747    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867750    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867753    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 13:33:09.871561 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867755    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867758    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867760    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867763    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867765    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867768    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867776    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867780    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867785    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867788    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867790    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867793    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867796    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867798    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867801    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867803    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867806    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867810    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867813    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 13:33:09.871945 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867816    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867819    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867822    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867825    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867828    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867831    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867833    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867836    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867839    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867841    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867843    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867846    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867848    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867851    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867854    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867856    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867859    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867861    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867865    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867868    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 13:33:09.872438 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867871    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867873    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867883    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867886    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867889    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867891    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867894    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867897    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867900    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867903    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867906    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867908    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867911    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867914    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867917    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867919    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867922    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867925    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867927    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867930    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 13:33:09.872921 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867933    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867935    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867940    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867943    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867946    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867949    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867953    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867956    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867958    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867961    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867964    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867968    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867971    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867973    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867976    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867979    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867981    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867984    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867986    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 13:33:09.873426 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867989    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867991    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867994    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867996    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.867999    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.868002    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.868004    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:09.868007    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.868012    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.868159    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.871402    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.872564    2556 server.go:1019] "Starting client certificate rotation"
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.872668    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 13:33:09.873884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.872708    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 13:33:09.875592 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.875578    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 13:33:09.876711 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.876693    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 13:33:09.886559 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.886539    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 16 13:33:09.889967 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.889952    2556 log.go:25] "Validated CRI v1 image API"
Sep 16 13:33:09.891627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.891602    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 13:33:09.895140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.895115    2556 fs.go:135] Filesystem UUIDs: map[4deec0d5-46fd-4b01-88ea-2ed4295f9762:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cae218e5-1e1b-4d4d-bd5d-e8998963c8e9:/dev/nvme0n1p4]
Sep 16 13:33:09.895140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.895134    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 13:33:09.900221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.900113    2556 manager.go:216] Machine: {Timestamp:2026-09-16 13:33:09.898676694 +0000 UTC m=+0.364613604 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089944 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2435a505108e36a4e7d012abdaed83 SystemUUID:ec2435a5-0510-8e36-a4e7-d012abdaed83 BootID:8757ae0b-261e-4ee2-8440-7c83e95bdabb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d1:b3:b1:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d1:b3:b1:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:58:cb:6c:fc:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 13:33:09.900221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.900215    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 13:33:09.900335 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.900291    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 13:33:09.903655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.903632    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 13:33:09.903795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.903659    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 13:33:09.904231 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.904216    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 13:33:09.904470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.904458    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 13:33:09.904502 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.904473    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 13:33:09.904502 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.904481    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 13:33:09.905359 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.905349    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 13:33:09.906657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.906646    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 13:33:09.910222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910210    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 13:33:09.910355 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910346    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 13:33:09.910390 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910376    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 13:33:09.910390 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910386    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 13:33:09.910440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910401    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 13:33:09.910440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910412    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 16 13:33:09.910440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.910421    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 13:33:09.913311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.913289    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 13:33:09.913311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.913316    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 13:33:09.915757 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.915739    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 13:33:09.918637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.918622    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 13:33:09.918693 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.918654    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 13:33:09.920796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920783    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 13:33:09.920796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920799    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920806    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920811    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920819    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920827    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920843    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920849    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920856    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920862    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920871    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 13:33:09.920904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.920876    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 13:33:09.921783 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.921770    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 13:33:09.921817 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.921784    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 13:33:09.922301 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.922276    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 13:33:09.922301 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.922277    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 13:33:09.925596 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.925580    2556 server.go:1268] "Started kubelet"
Sep 16 13:33:09.925754 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.925691    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 13:33:09.925754 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.925737    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 13:33:09.925848 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.925818    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 13:33:09.926540 ip-10-0-133-251 systemd[1]: Started Kubernetes Kubelet.
Sep 16 13:33:09.927587 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.927496    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 13:33:09.927868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.927845    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66g6r"
Sep 16 13:33:09.927958 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.927931    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 13:33:09.929849 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.929831    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 13:33:09.933126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.933112    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 13:33:09.933313 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.933301    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66g6r"
Sep 16 13:33:09.934377 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.933244    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-251.ec2.internal.18d5d06c29c65f29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-251.ec2.internal,UID:ip-10-0-133-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-251.ec2.internal,},FirstTimestamp:2026-09-16 13:33:09.925551913 +0000 UTC m=+0.391488812,LastTimestamp:2026-09-16 13:33:09.925551913 +0000 UTC m=+0.391488812,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-251.ec2.internal,}"
Sep 16 13:33:09.935978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.935961    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 13:33:09.936678 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.936662    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 13:33:09.936791 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.936771    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:09.937440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937426    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 13:33:09.937440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937444    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 13:33:09.937548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937469    2556 factory.go:55] Registering systemd factory
Sep 16 13:33:09.937548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937482    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 16 13:33:09.937548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937425    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 13:33:09.937718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937702    2556 factory.go:153] Registering CRI-O factory
Sep 16 13:33:09.937718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937718    2556 factory.go:223] Registration of the crio container factory successfully
Sep 16 13:33:09.937836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937768    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 13:33:09.937836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937788    2556 factory.go:103] Registering Raw factory
Sep 16 13:33:09.937836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.937802    2556 manager.go:1196] Started watching for new ooms in manager
Sep 16 13:33:09.938102 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.938066    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 13:33:09.938165 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.938134    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 13:33:09.938216 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.938202    2556 manager.go:349] Starting recovery of all containers
Sep 16 13:33:09.938307 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.938286    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 13:33:09.938388 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.938368    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 13:33:09.949866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.949713    2556 manager.go:354] Recovery completed
Sep 16 13:33:09.954446 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.954431    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:09.956207 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956191    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:09.956269 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956220    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:09.956269 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956231    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:09.956753 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956736    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 13:33:09.956753 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956752    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 13:33:09.956900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956768    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 13:33:09.956900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.956810    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:09.957434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.957420    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:09.957492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.957446    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:09.957492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.957456    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:09.957968 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.957955    2556 policy_none.go:49] "None policy: Start"
Sep 16 13:33:09.958013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.957975    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 13:33:09.958013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.957988    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 13:33:09.958985 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.958975    2556 policy_none.go:47] "Start"
Sep 16 13:33:09.989700 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.989681    2556 manager.go:337] "Starting Device Plugin manager"
Sep 16 13:33:09.989793 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.989728    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 13:33:09.989793 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.989741    2556 server.go:87] "Starting device plugin registration server"
Sep 16 13:33:09.990058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.990047    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 13:33:09.990113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.990060    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 13:33:09.990210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.990191    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 13:33:09.990313 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.990278    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 13:33:09.990313 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:09.990292    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 13:33:09.990716 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.990683    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 13:33:09.990791 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:09.990745    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.050179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.050134    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 13:33:10.051922 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.051897    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 13:33:10.052019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.051950    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 13:33:10.052019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.051976    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 13:33:10.052125 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.052025    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 13:33:10.054304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.054286    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:33:10.091170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.091139    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:10.092050 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.092038    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:10.092129 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.092068    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:10.092129 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.092082    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:10.092129 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.092127    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.100550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.100472    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.100550 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.100495    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-251.ec2.internal\": node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.117525 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.117502    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.153036 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.153007    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"]
Sep 16 13:33:10.153152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.153077    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:10.154538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.154522    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:10.154602 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.154554    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:10.154602 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.154571    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:10.155807 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.155793    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:10.155940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.155927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.155976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.155955    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:10.156556 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.156542    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:10.156556 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.156548    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:10.156658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.156568    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:10.156658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.156569    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:10.156658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.156577    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:10.156658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.156581    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:10.157578 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.157565    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.157646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.157599    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 13:33:10.158210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.158194    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 13:33:10.158301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.158218    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 13:33:10.158301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.158228    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeHasSufficientPID"
Sep 16 13:33:10.186861 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.186836    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-251.ec2.internal\" not found" node="ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.190345 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.190329    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-251.ec2.internal\" not found" node="ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.218034 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.218011    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.239118 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.239077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95e0be1d4166ccb5c95018175f7c6e1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"95e0be1d4166ccb5c95018175f7c6e1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.239252 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.239124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95e0be1d4166ccb5c95018175f7c6e1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"95e0be1d4166ccb5c95018175f7c6e1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.239252 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.239152    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dc59d1cdb7714ddbcf795e023ef2c6a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"7dc59d1cdb7714ddbcf795e023ef2c6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.319049 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.319013    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.339441 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.339416    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dc59d1cdb7714ddbcf795e023ef2c6a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"7dc59d1cdb7714ddbcf795e023ef2c6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.339541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.339446    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95e0be1d4166ccb5c95018175f7c6e1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"95e0be1d4166ccb5c95018175f7c6e1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.339541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.339468    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95e0be1d4166ccb5c95018175f7c6e1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"95e0be1d4166ccb5c95018175f7c6e1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.339541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.339504    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dc59d1cdb7714ddbcf795e023ef2c6a-config\") pod \"kube-apiserver-proxy-ip-10-0-133-251.ec2.internal\" (UID: \"7dc59d1cdb7714ddbcf795e023ef2c6a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.339541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.339536    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95e0be1d4166ccb5c95018175f7c6e1a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"95e0be1d4166ccb5c95018175f7c6e1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.339698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.339507    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95e0be1d4166ccb5c95018175f7c6e1a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal\" (UID: \"95e0be1d4166ccb5c95018175f7c6e1a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.419832 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.419803    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.489280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.489240    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.492841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.492776    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.520341 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.520312    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.620926 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.620884    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.721504 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.721442    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.821937 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.821909    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-251.ec2.internal\" not found"
Sep 16 13:33:10.859544 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.859524    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:33:10.872466 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.872450    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 13:33:10.872560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.872544    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 13:33:10.872610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.872592    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 13:33:10.911710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.911688    2556 apiserver.go:52] "Watching apiserver"
Sep 16 13:33:10.927489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.927458    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 13:33:10.927812 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.927792    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cxlj6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt","openshift-dns/node-resolver-52d77","openshift-multus/multus-additional-cni-plugins-x92sh","openshift-multus/network-metrics-daemon-8w2s9","openshift-network-operator/iptables-alerter-k8qmv","openshift-ovn-kubernetes/ovnkube-node-2j9wp","kube-system/konnectivity-agent-rvzwh","openshift-cluster-node-tuning-operator/tuned-mj7jz","openshift-image-registry/node-ca-vp7zm","openshift-multus/multus-vkg5d","openshift-network-diagnostics/network-check-target-9gskl"]
Sep 16 13:33:10.929176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.929159    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:10.929260 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.929238    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:10.930263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.930244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.931376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.931359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:10.932170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932146    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 13:33:10.932264 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932150    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.932264 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932220    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8hzzp\""
Sep 16 13:33:10.932264 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932220    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.932428 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932404    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.932844 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932828    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.932945 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932932    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.933001 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.932979    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s69xn\""
Sep 16 13:33:10.933383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.933369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:10.933449 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.933418    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:10.933967 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.933953    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.934340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.934325    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.934537 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.934523    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:10.934603 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.934535    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 13:33:10.934603 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.934577    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 13:33:10.934791 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.934774    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 13:33:10.934931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.934917    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9j9w\""
Sep 16 13:33:10.935835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.935803    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 13:28:10 +0000 UTC" deadline="2028-06-28 03:46:32.998004486 +0000 UTC"
Sep 16 13:33:10.935888 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.935835    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15614h13m22.062171236s"
Sep 16 13:33:10.936134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936123    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.936186 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936141    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 13:33:10.936536 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936520    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.936615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936543    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 13:33:10.936615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936564    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hhwv6\""
Sep 16 13:33:10.936739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936726    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.936846 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.936833    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.937627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.937612    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:10.938185 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938172    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 13:33:10.938185 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938179    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.938538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938510    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 13:33:10.938645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938560    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 13:33:10.938645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938566    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.938645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938573    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dptsm\""
Sep 16 13:33:10.938645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938592    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 13:33:10.938911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.938898    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.939708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.939691    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-59qss\""
Sep 16 13:33:10.939785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.939764    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 13:33:10.939785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.939766    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 13:33:10.940120 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.940107    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:10.940455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.940439    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.940657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.940640    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.940705 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.940640    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7v6qh\""
Sep 16 13:33:10.941297 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.941283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:10.941615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.941600    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 13:33:10.941682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.941664    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 13:33:10.941903 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.941887    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 13:33:10.941987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.941973    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mgtx9\""
Sep 16 13:33:10.942376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942360    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:10.942449 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.942449 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942409    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.942449 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:10.942415    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:10.942449 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942431    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysctl-d\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-kubernetes\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942477    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-var-lib-kubelet\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmsd\" (UniqueName: \"kubernetes.io/projected/14ab078d-52b2-409b-b8e8-0e517825a6e7-kube-api-access-chmsd\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-node-log\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-env-overrides\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942569    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-modprobe-d\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-host\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.942617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942616    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-etc-selinux\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.942933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mb6\" (UniqueName: \"kubernetes.io/projected/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-kube-api-access-q7mb6\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.942933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-slash\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.942933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-lib-modules\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.942933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkxqf\" (UniqueName: \"kubernetes.io/projected/2786ffeb-964f-42fe-995f-e3be60ecd47f-kube-api-access-nkxqf\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:10.942933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-ovn\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.943190 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.942809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54d8f078-4001-4193-8c85-2951ebc88278-kubelet-config\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:10.943260 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943236    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9-agent-certs\") pod \"konnectivity-agent-rvzwh\" (UID: \"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9\") " pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:10.943317 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysconfig\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.943374 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943359    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-run\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943501    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943631    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-system-cni-dir\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-cni-netd\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccd06b3d-618d-4cab-950d-f86fe6bf7912-tmp\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.943930    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cnibin\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-cni-bin\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944026    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-t8ll9\""
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-systemd\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-sys\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-device-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb9lv\" (UniqueName: \"kubernetes.io/projected/579cf366-9e9d-4bb1-b00f-2fb3270d5115-kube-api-access-sb9lv\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944359    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysctl-conf\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.944511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944410    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27850248-ce2d-480c-83f7-05c2477fb14f-tmp-dir\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-kubelet-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944591    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-systemd-units\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944642    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovnkube-config\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-sys-fs\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-systemd\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944804    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-var-lib-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944857    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgckm\" (UniqueName: \"kubernetes.io/projected/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-kube-api-access-cgckm\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54d8f078-4001-4193-8c85-2951ebc88278-dbus\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nvlh\" (UniqueName: \"kubernetes.io/projected/27850248-ce2d-480c-83f7-05c2477fb14f-kube-api-access-6nvlh\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.944969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-tuned\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-socket-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945044    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovn-node-metrics-cert\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945108    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovnkube-script-lib\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-registration-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-etc-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9-konnectivity-ca\") pod \"konnectivity-agent-rvzwh\" (UID: \"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9\") " pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945247    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmhd\" (UniqueName: \"kubernetes.io/projected/ccd06b3d-618d-4cab-950d-f86fe6bf7912-kube-api-access-8gmhd\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945284    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27850248-ce2d-480c-83f7-05c2477fb14f-hosts-file\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-kubelet\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-run-netns\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945363    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-log-socket\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945535    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14ab078d-52b2-409b-b8e8-0e517825a6e7-iptables-alerter-script\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:10.945970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.945620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14ab078d-52b2-409b-b8e8-0e517825a6e7-host-slash\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:10.947020 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.946973    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 13:33:10.947127 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.947112    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"
Sep 16 13:33:10.947353 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.947336    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal"]
Sep 16 13:33:10.950953 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.950938    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 13:33:10.952579 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.952566    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal"]
Sep 16 13:33:10.952644 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.952630    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 13:33:10.967043 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.967026    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6b6f7"
Sep 16 13:33:10.973518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.973480    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6b6f7"
Sep 16 13:33:10.987061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:10.987042    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:33:11.038830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.038811    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 13:33:11.045983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.045964    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkxqf\" (UniqueName: \"kubernetes.io/projected/2786ffeb-964f-42fe-995f-e3be60ecd47f-kube-api-access-nkxqf\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:11.046061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.045992    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-ovn\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.046061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54d8f078-4001-4193-8c85-2951ebc88278-kubelet-config\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:11.046061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-netns\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046070    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-ovn\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/54d8f078-4001-4193-8c85-2951ebc88278-kubelet-config\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046123    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9-agent-certs\") pod \"konnectivity-agent-rvzwh\" (UID: \"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9\") " pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysconfig\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-run\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046228    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-system-cni-dir\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.046262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-system-cni-dir\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-cni-netd\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046305    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-cni-netd\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046258    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-run\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046230    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysconfig\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046304    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccd06b3d-618d-4cab-950d-f86fe6bf7912-tmp\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da85859e-800e-4d7c-8b95-775d36698fdd-serviceca\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cnibin\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-cni-bin\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046426    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-systemd\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046450    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-sys\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cnibin\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046462    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-cni-bin\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046469    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-systemd\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046474    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-device-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046492    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-sys\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.046583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046504    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb9lv\" (UniqueName: \"kubernetes.io/projected/579cf366-9e9d-4bb1-b00f-2fb3270d5115-kube-api-access-sb9lv\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046551    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysctl-conf\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046567    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27850248-ce2d-480c-83f7-05c2477fb14f-tmp-dir\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046590    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-kubelet-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046627    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-systemd-units\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovnkube-config\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysctl-conf\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046674    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-system-cni-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-kubelet-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046602    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-device-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-cni-bin\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046723    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-systemd-units\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a32af615-2666-4782-807f-c7fa822b8706-multus-daemon-config\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.047481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046822    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-sys-fs\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-systemd\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046867    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-sys-fs\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-var-lib-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046891    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27850248-ce2d-480c-83f7-05c2477fb14f-tmp-dir\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-systemd\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046913    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgckm\" (UniqueName: \"kubernetes.io/projected/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-kube-api-access-cgckm\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046925    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-var-lib-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54d8f078-4001-4193-8c85-2951ebc88278-dbus\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-conf-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.046996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nvlh\" (UniqueName: \"kubernetes.io/projected/27850248-ce2d-480c-83f7-05c2477fb14f-kube-api-access-6nvlh\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-tuned\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047030    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/54d8f078-4001-4193-8c85-2951ebc88278-dbus\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-cni-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047083    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-cnibin\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-socket-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovn-node-metrics-cert\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047199    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovnkube-script-lib\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-etc-kubernetes\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-registration-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047316    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-etc-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-registration-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047325    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047349    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9-konnectivity-ca\") pod \"konnectivity-agent-rvzwh\" (UID: \"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9\") " pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-etc-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047382    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-socket-dir\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmhd\" (UniqueName: \"kubernetes.io/projected/ccd06b3d-618d-4cab-950d-f86fe6bf7912-kube-api-access-8gmhd\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-kubelet\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047802    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-hostroot\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-multus-certs\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27850248-ce2d-480c-83f7-05c2477fb14f-hosts-file\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-kubelet\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.047969    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-run-netns\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.048917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-log-socket\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048058    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-k8s-cni-cncf-io\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52gcd\" (UniqueName: \"kubernetes.io/projected/a32af615-2666-4782-807f-c7fa822b8706-kube-api-access-52gcd\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048202    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048228    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14ab078d-52b2-409b-b8e8-0e517825a6e7-iptables-alerter-script\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048258    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14ab078d-52b2-409b-b8e8-0e517825a6e7-host-slash\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048272    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovnkube-script-lib\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.048285    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da85859e-800e-4d7c-8b95-775d36698fdd-host\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovnkube-config\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.049745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049637    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysctl-d\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049768    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-cni-multus\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-kubernetes\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-var-lib-kubelet\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32af615-2666-4782-807f-c7fa822b8706-cni-binary-copy\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88n6\" (UniqueName: \"kubernetes.io/projected/da85859e-800e-4d7c-8b95-775d36698fdd-kube-api-access-p88n6\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmsd\" (UniqueName: \"kubernetes.io/projected/14ab078d-52b2-409b-b8e8-0e517825a6e7-kube-api-access-chmsd\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.049982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-node-log\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-env-overrides\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-modprobe-d\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-host\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-socket-dir-parent\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-etc-selinux\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mb6\" (UniqueName: \"kubernetes.io/projected/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-kube-api-access-q7mb6\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050241    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-slash\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050274    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-lib-modules\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:11.050610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.051464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-sysctl-d\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.051464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050645    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-kubernetes\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.051464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.050702    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-var-lib-kubelet\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.051464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.051031    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-node-log\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.051464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.051445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-env-overrides\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.051674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.051551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-modprobe-d\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.051674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.051612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-host\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.051787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.051681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/579cf366-9e9d-4bb1-b00f-2fb3270d5115-etc-selinux\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.051989 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.051962    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-slash\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.052123 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.052103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ccd06b3d-618d-4cab-950d-f86fe6bf7912-lib-modules\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.052197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.052134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27850248-ce2d-480c-83f7-05c2477fb14f-hosts-file\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.052197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.052182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-kubelet\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.052331 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.052238    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-host-run-netns\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.052331 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.052281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-run-openvswitch\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.052445 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.052328    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-log-socket\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.052445 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.052429    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:11.052548 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.052528    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:11.552490641 +0000 UTC m=+2.018427529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:11.052825 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.052806    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:11.052886 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.052868    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:33:11.552845275 +0000 UTC m=+2.018782166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:11.055318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.054825    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.055318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.055102    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14ab078d-52b2-409b-b8e8-0e517825a6e7-host-slash\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.055318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.055196    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.055540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.055460    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/14ab078d-52b2-409b-b8e8-0e517825a6e7-iptables-alerter-script\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.055540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.055530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.056109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.055915    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9-konnectivity-ca\") pod \"konnectivity-agent-rvzwh\" (UID: \"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9\") " pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:11.056894 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.056141    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkxqf\" (UniqueName: \"kubernetes.io/projected/2786ffeb-964f-42fe-995f-e3be60ecd47f-kube-api-access-nkxqf\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:11.056894 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.056691    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-ovn-node-metrics-cert\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.056894 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.056794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9-agent-certs\") pod \"konnectivity-agent-rvzwh\" (UID: \"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9\") " pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:11.057675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.057301    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccd06b3d-618d-4cab-950d-f86fe6bf7912-tmp\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.057675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.057621    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb9lv\" (UniqueName: \"kubernetes.io/projected/579cf366-9e9d-4bb1-b00f-2fb3270d5115-kube-api-access-sb9lv\") pod \"aws-ebs-csi-driver-node-872gt\" (UID: \"579cf366-9e9d-4bb1-b00f-2fb3270d5115\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.057675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.057664    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nvlh\" (UniqueName: \"kubernetes.io/projected/27850248-ce2d-480c-83f7-05c2477fb14f-kube-api-access-6nvlh\") pod \"node-resolver-52d77\" (UID: \"27850248-ce2d-480c-83f7-05c2477fb14f\") " pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.057832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.057711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgckm\" (UniqueName: \"kubernetes.io/projected/0ac88517-88f4-49f8-b15f-aa8d37c14c3f-kube-api-access-cgckm\") pod \"ovnkube-node-2j9wp\" (UID: \"0ac88517-88f4-49f8-b15f-aa8d37c14c3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.058524 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.058500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ccd06b3d-618d-4cab-950d-f86fe6bf7912-etc-tuned\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.058668 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.058650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmsd\" (UniqueName: \"kubernetes.io/projected/14ab078d-52b2-409b-b8e8-0e517825a6e7-kube-api-access-chmsd\") pod \"iptables-alerter-k8qmv\" (UID: \"14ab078d-52b2-409b-b8e8-0e517825a6e7\") " pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.059730 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.059712    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mb6\" (UniqueName: \"kubernetes.io/projected/0c5a16d7-8099-4cf5-a759-1684dbbc46a0-kube-api-access-q7mb6\") pod \"multus-additional-cni-plugins-x92sh\" (UID: \"0c5a16d7-8099-4cf5-a759-1684dbbc46a0\") " pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.061426 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.061403    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmhd\" (UniqueName: \"kubernetes.io/projected/ccd06b3d-618d-4cab-950d-f86fe6bf7912-kube-api-access-8gmhd\") pod \"tuned-mj7jz\" (UID: \"ccd06b3d-618d-4cab-950d-f86fe6bf7912\") " pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.071966 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.071950    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mj7jz"
Sep 16 13:33:11.130677 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.130653    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:33:11.151446 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151422    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-system-cni-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-cni-bin\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a32af615-2666-4782-807f-c7fa822b8706-multus-daemon-config\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-conf-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-cni-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-cnibin\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151546    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-system-cni-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151571    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-conf-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151576    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-etc-kubernetes\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-etc-kubernetes\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151618    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-cnibin\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151546    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-cni-bin\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-kubelet\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-cni-dir\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-hostroot\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.151694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151694    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-kubelet\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151697    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-multus-certs\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151725    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-multus-certs\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-k8s-cni-cncf-io\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-hostroot\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151773    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52gcd\" (UniqueName: \"kubernetes.io/projected/a32af615-2666-4782-807f-c7fa822b8706-kube-api-access-52gcd\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151805    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-k8s-cni-cncf-io\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da85859e-800e-4d7c-8b95-775d36698fdd-host\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151835    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-cni-multus\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151846    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da85859e-800e-4d7c-8b95-775d36698fdd-host\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151861    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32af615-2666-4782-807f-c7fa822b8706-cni-binary-copy\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151869    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-var-lib-cni-multus\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88n6\" (UniqueName: \"kubernetes.io/projected/da85859e-800e-4d7c-8b95-775d36698fdd-kube-api-access-p88n6\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151924    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-socket-dir-parent\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151950    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.151981    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-netns\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.152003    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-multus-socket-dir-parent\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.152023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da85859e-800e-4d7c-8b95-775d36698fdd-serviceca\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.152273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.152052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a32af615-2666-4782-807f-c7fa822b8706-host-run-netns\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.152069    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a32af615-2666-4782-807f-c7fa822b8706-multus-daemon-config\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.152294    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a32af615-2666-4782-807f-c7fa822b8706-cni-binary-copy\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.152866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.152481    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da85859e-800e-4d7c-8b95-775d36698fdd-serviceca\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.156745 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.156729    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:11.156801 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.156749    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:11.156801 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.156759    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:11.156864 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.156807    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:11.656793656 +0000 UTC m=+2.122730560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:11.159988 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.159972    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52gcd\" (UniqueName: \"kubernetes.io/projected/a32af615-2666-4782-807f-c7fa822b8706-kube-api-access-52gcd\") pod \"multus-vkg5d\" (UID: \"a32af615-2666-4782-807f-c7fa822b8706\") " pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.160106 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.160072    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88n6\" (UniqueName: \"kubernetes.io/projected/da85859e-800e-4d7c-8b95-775d36698fdd-kube-api-access-p88n6\") pod \"node-ca-vp7zm\" (UID: \"da85859e-800e-4d7c-8b95-775d36698fdd\") " pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.260215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.260135    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt"
Sep 16 13:33:11.273864 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.273843    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52d77"
Sep 16 13:33:11.289695 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.289667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-x92sh"
Sep 16 13:33:11.301466 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.301424    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5a16d7_8099_4cf5_a759_1684dbbc46a0.slice/crio-212b7908d5253268316c4e7141d061a7799b2ac4320c85851550955f9ebb3723 WatchSource:0}: Error finding container 212b7908d5253268316c4e7141d061a7799b2ac4320c85851550955f9ebb3723: Status 404 returned error can't find the container with id 212b7908d5253268316c4e7141d061a7799b2ac4320c85851550955f9ebb3723
Sep 16 13:33:11.301988 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.301965    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e0be1d4166ccb5c95018175f7c6e1a.slice/crio-bb17844d3965630cec9a03b7cb03aabd917e824bf6b11ba88a861640f6555287 WatchSource:0}: Error finding container bb17844d3965630cec9a03b7cb03aabd917e824bf6b11ba88a861640f6555287: Status 404 returned error can't find the container with id bb17844d3965630cec9a03b7cb03aabd917e824bf6b11ba88a861640f6555287
Sep 16 13:33:11.302505 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.302485    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27850248_ce2d_480c_83f7_05c2477fb14f.slice/crio-dba85cb1506481c384ebe41141fb2c5374c3d8cf17736cba7020eb4d0adfcb8d WatchSource:0}: Error finding container dba85cb1506481c384ebe41141fb2c5374c3d8cf17736cba7020eb4d0adfcb8d: Status 404 returned error can't find the container with id dba85cb1506481c384ebe41141fb2c5374c3d8cf17736cba7020eb4d0adfcb8d
Sep 16 13:33:11.306381 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.306361    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:33:11.311102 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.311065    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8qmv"
Sep 16 13:33:11.318762 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.318741    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ab078d_52b2_409b_b8e8_0e517825a6e7.slice/crio-06a7cac09160a91bb21c55dfb8dafb130380f3b911e155241d9e5e7105134624 WatchSource:0}: Error finding container 06a7cac09160a91bb21c55dfb8dafb130380f3b911e155241d9e5e7105134624: Status 404 returned error can't find the container with id 06a7cac09160a91bb21c55dfb8dafb130380f3b911e155241d9e5e7105134624
Sep 16 13:33:11.328443 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.328427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:11.337166 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.337146    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac88517_88f4_49f8_b15f_aa8d37c14c3f.slice/crio-47a2d3ba16afb4d0247a1a79d3a83c259abd45e5a2d6257f209189f659387851 WatchSource:0}: Error finding container 47a2d3ba16afb4d0247a1a79d3a83c259abd45e5a2d6257f209189f659387851: Status 404 returned error can't find the container with id 47a2d3ba16afb4d0247a1a79d3a83c259abd45e5a2d6257f209189f659387851
Sep 16 13:33:11.350494 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.350474    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:11.357621 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.357598    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fb372c_6db1_43f6_9e4d_094f9cf7b9e9.slice/crio-bfc4484a10754b6bb0c699dfad7b3e00b4a308b37d55882862f125fbe7c35805 WatchSource:0}: Error finding container bfc4484a10754b6bb0c699dfad7b3e00b4a308b37d55882862f125fbe7c35805: Status 404 returned error can't find the container with id bfc4484a10754b6bb0c699dfad7b3e00b4a308b37d55882862f125fbe7c35805
Sep 16 13:33:11.384972 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.384949    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vp7zm"
Sep 16 13:33:11.390118 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.390042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkg5d"
Sep 16 13:33:11.392412 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.392390    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda85859e_800e_4d7c_8b95_775d36698fdd.slice/crio-deba73c68e7a42e860c3d261a2671fa2eedee8e844a9ac628827d4db8868f8d9 WatchSource:0}: Error finding container deba73c68e7a42e860c3d261a2671fa2eedee8e844a9ac628827d4db8868f8d9: Status 404 returned error can't find the container with id deba73c68e7a42e860c3d261a2671fa2eedee8e844a9ac628827d4db8868f8d9
Sep 16 13:33:11.397565 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.397545    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32af615_2666_4782_807f_c7fa822b8706.slice/crio-c659f32dc2e35c2ef842943379d4d08f9df2a8fa699ee813c4b9593715269792 WatchSource:0}: Error finding container c659f32dc2e35c2ef842943379d4d08f9df2a8fa699ee813c4b9593715269792: Status 404 returned error can't find the container with id c659f32dc2e35c2ef842943379d4d08f9df2a8fa699ee813c4b9593715269792
Sep 16 13:33:11.479348 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.479317    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd06b3d_618d_4cab_950d_f86fe6bf7912.slice/crio-2c364324e54425c662af3f87af502da35bbfd8b46959e439600909f423e3d2cf WatchSource:0}: Error finding container 2c364324e54425c662af3f87af502da35bbfd8b46959e439600909f423e3d2cf: Status 404 returned error can't find the container with id 2c364324e54425c662af3f87af502da35bbfd8b46959e439600909f423e3d2cf
Sep 16 13:33:11.555180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.555155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:11.555263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.555190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:11.555340 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.555326    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:11.555390 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.555328    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:11.555430 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.555391    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:12.555370732 +0000 UTC m=+3.021307620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:11.555430 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.555412    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:33:12.555401351 +0000 UTC m=+3.021338238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:11.629280 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:11.629247    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579cf366_9e9d_4bb1_b00f_2fb3270d5115.slice/crio-179c65449df2c55c72905d641327dbff0145eb07dc535fbf1851f786ee4a7019 WatchSource:0}: Error finding container 179c65449df2c55c72905d641327dbff0145eb07dc535fbf1851f786ee4a7019: Status 404 returned error can't find the container with id 179c65449df2c55c72905d641327dbff0145eb07dc535fbf1851f786ee4a7019
Sep 16 13:33:11.757299 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.757081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:11.757500 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.757354    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:11.757500 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.757378    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:11.757500 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.757393    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:11.757500 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:11.757481    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:12.757445435 +0000 UTC m=+3.223382322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:11.975404 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.975312    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:28:11 +0000 UTC" deadline="2028-02-11 08:32:32.754730456 +0000 UTC"
Sep 16 13:33:11.975404 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:11.975351    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12306h59m20.77938494s"
Sep 16 13:33:12.054992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.054966    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:12.055201 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.055091    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:12.055595 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.055564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:12.055723 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.055689    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:12.059510 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.059443    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mj7jz" event={"ID":"ccd06b3d-618d-4cab-950d-f86fe6bf7912","Type":"ContainerStarted","Data":"2c364324e54425c662af3f87af502da35bbfd8b46959e439600909f423e3d2cf"}
Sep 16 13:33:12.062378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.062317    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkg5d" event={"ID":"a32af615-2666-4782-807f-c7fa822b8706","Type":"ContainerStarted","Data":"c659f32dc2e35c2ef842943379d4d08f9df2a8fa699ee813c4b9593715269792"}
Sep 16 13:33:12.064615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.064572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8qmv" event={"ID":"14ab078d-52b2-409b-b8e8-0e517825a6e7","Type":"ContainerStarted","Data":"06a7cac09160a91bb21c55dfb8dafb130380f3b911e155241d9e5e7105134624"}
Sep 16 13:33:12.066490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.066464    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"95e0be1d4166ccb5c95018175f7c6e1a","Type":"ContainerStarted","Data":"bb17844d3965630cec9a03b7cb03aabd917e824bf6b11ba88a861640f6555287"}
Sep 16 13:33:12.068528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.068503    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52d77" event={"ID":"27850248-ce2d-480c-83f7-05c2477fb14f","Type":"ContainerStarted","Data":"dba85cb1506481c384ebe41141fb2c5374c3d8cf17736cba7020eb4d0adfcb8d"}
Sep 16 13:33:12.070772 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.070747    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt" event={"ID":"579cf366-9e9d-4bb1-b00f-2fb3270d5115","Type":"ContainerStarted","Data":"179c65449df2c55c72905d641327dbff0145eb07dc535fbf1851f786ee4a7019"}
Sep 16 13:33:12.074010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.073981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vp7zm" event={"ID":"da85859e-800e-4d7c-8b95-775d36698fdd","Type":"ContainerStarted","Data":"deba73c68e7a42e860c3d261a2671fa2eedee8e844a9ac628827d4db8868f8d9"}
Sep 16 13:33:12.077117 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.077082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rvzwh" event={"ID":"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9","Type":"ContainerStarted","Data":"bfc4484a10754b6bb0c699dfad7b3e00b4a308b37d55882862f125fbe7c35805"}
Sep 16 13:33:12.079999 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.079975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"47a2d3ba16afb4d0247a1a79d3a83c259abd45e5a2d6257f209189f659387851"}
Sep 16 13:33:12.083723 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.083696    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerStarted","Data":"212b7908d5253268316c4e7141d061a7799b2ac4320c85851550955f9ebb3723"}
Sep 16 13:33:12.252779 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:12.252739    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc59d1cdb7714ddbcf795e023ef2c6a.slice/crio-fd06143723c59c3379352d9ca3e9b8649a0405cd1fdfcbbeedfe1cc6806f8973 WatchSource:0}: Error finding container fd06143723c59c3379352d9ca3e9b8649a0405cd1fdfcbbeedfe1cc6806f8973: Status 404 returned error can't find the container with id fd06143723c59c3379352d9ca3e9b8649a0405cd1fdfcbbeedfe1cc6806f8973
Sep 16 13:33:12.331807 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.331768    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:33:12.566549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.566460    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:12.566549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.566512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:12.566798 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.566665    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:12.566798 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.566725    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:33:14.566705426 +0000 UTC m=+5.032642317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:12.567159 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.567136    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:12.567282 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.567217    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:14.567197054 +0000 UTC m=+5.033133945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:12.769642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.769601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:12.769857 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.769809    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:12.769857 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.769835    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:12.769857 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.769850    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:12.770031 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:12.769915    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:14.769896228 +0000 UTC m=+5.235833119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:12.976366 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.976292    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 13:28:11 +0000 UTC" deadline="2028-06-12 10:47:27.007880966 +0000 UTC"
Sep 16 13:33:12.976366 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:12.976364    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15237h14m14.031522215s"
Sep 16 13:33:13.052732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:13.052191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:13.052732 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:13.052350    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:13.125635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:13.125509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" event={"ID":"7dc59d1cdb7714ddbcf795e023ef2c6a","Type":"ContainerStarted","Data":"fd06143723c59c3379352d9ca3e9b8649a0405cd1fdfcbbeedfe1cc6806f8973"}
Sep 16 13:33:13.846387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:13.846352    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 13:33:14.062053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:14.062019    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:14.062566 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.062193    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:14.062566 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:14.062288    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:14.062566 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.062369    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:14.587336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:14.586445    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:14.587336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:14.586506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:14.587336 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.586715    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:14.587336 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.586787    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:33:18.586765288 +0000 UTC m=+9.052702185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:14.587336 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.587238    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:14.587336 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.587293    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:18.587276833 +0000 UTC m=+9.053213723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:14.788728 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:14.787975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:14.788728 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.788229    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:14.788728 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.788254    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:14.788728 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.788269    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:14.788728 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:14.788334    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:18.788314913 +0000 UTC m=+9.254251806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:15.052504 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:15.052464    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:15.052725 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:15.052614    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:16.055987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:16.055961    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:16.056509 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:16.056073    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:16.056509 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:16.056392    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:16.056509 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:16.056464    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:17.052462 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:17.052427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:17.052679 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:17.052572    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:18.052310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:18.052271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:18.052826 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.052411    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:18.052826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:18.052271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:18.052978 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.052851    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:18.624850 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:18.624800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:18.625044 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:18.624861    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:18.625044 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.625036    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:18.625219 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.625107    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:33:26.625084772 +0000 UTC m=+17.091021668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:18.625608 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.625585    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:18.625698 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.625647    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:26.625629624 +0000 UTC m=+17.091566514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:18.826548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:18.826504    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:18.826769 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.826722    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:18.826769 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.826755    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:18.826769 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.826771    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:18.826946 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:18.826839    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:26.826818341 +0000 UTC m=+17.292755235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:19.052289 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:19.052257    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:19.052498 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:19.052399    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:20.053661 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:20.053624    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:20.054184 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:20.053678    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:20.054184 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:20.053753    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:20.054184 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:20.053831    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:21.052917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:21.052881    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:21.053131 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:21.053024    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:22.053250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:22.053216    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:22.053250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:22.053240    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:22.053778 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:22.053338    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:22.053778 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:22.053479    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:23.052563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:23.052529    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:23.052737 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:23.052659    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:24.052625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:24.052592    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:24.053068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:24.052601    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:24.053068 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:24.052705    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:24.053068 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:24.052829    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:25.052201 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:25.052166    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:25.052392 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:25.052305    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:26.053321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:26.053287    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:26.053692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:26.053293    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:26.053692 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.053406    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:26.053692 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.053510    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:26.686937 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:26.686907    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:26.687064 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:26.686944    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:26.687064 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.687052    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:26.687064 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.687053    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:26.687195 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.687121    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:33:42.687105924 +0000 UTC m=+33.153042810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:26.687195 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.687135    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:42.687128733 +0000 UTC m=+33.153065618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:26.889323 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:26.889234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:26.889502 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.889401    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:26.889502 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.889421    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:26.889502 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.889435    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:26.889502 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:26.889500    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:42.889481981 +0000 UTC m=+33.355418867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:27.053101 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:27.053048    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:27.053283 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:27.053169    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:28.052668 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:28.052630    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:28.053188 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:28.052634    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:28.053188 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:28.052787    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:28.053188 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:28.052836    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:29.052222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:29.052185    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:29.052401 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:29.052304    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:30.053080 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:30.053050    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:30.053577 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:30.053168    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:30.053577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:30.053282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:30.053577 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:30.053456    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:31.052447 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:31.052417    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:31.052638 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:31.052525    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:32.052670 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.052393    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:32.053277 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.052459    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:32.053277 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:32.052774    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:32.053277 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:32.052841    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:32.164746 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.164714    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt" event={"ID":"579cf366-9e9d-4bb1-b00f-2fb3270d5115","Type":"ContainerStarted","Data":"ea86df5278d4ee34c0b9a0d59ffbaf3d5616fb0fb264fbe0481e81cf7908651d"}
Sep 16 13:33:32.166033 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.166008    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vp7zm" event={"ID":"da85859e-800e-4d7c-8b95-775d36698fdd","Type":"ContainerStarted","Data":"f3be9fca7ac42aa1a8416dc37d01d433e38f672d6282f3af8484c2f7f8ede27b"}
Sep 16 13:33:32.167380 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.167358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rvzwh" event={"ID":"f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9","Type":"ContainerStarted","Data":"86f18110e84089cf8f759262eadc51f0433c2e0ac45059719cc6ecb167b12fb4"}
Sep 16 13:33:32.169699 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.169679    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"bab8de4b1f57db7c59c8a659b15da737b70767a4154010c6ef1dfae482a54c05"}
Sep 16 13:33:32.169794 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.169738    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"6e424aa5e167c0df8bcc113afc74bd0bf6c6e4e8586bfd3ab106572f81fffba3"}
Sep 16 13:33:32.169794 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.169774    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"87a7643a998c959ddbf5ebb9b159c8870c0d2eba14b601ebcbb878f66ebf24c2"}
Sep 16 13:33:32.169880 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.169845    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"8a520b4e113775d8c1565ba87e7e3047e438ee0b176e0a102317753a1f7895bb"}
Sep 16 13:33:32.171251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.171229    2556 generic.go:356] "Generic (PLEG): container finished" podID="0c5a16d7-8099-4cf5-a759-1684dbbc46a0" containerID="d0e600a49e585ac5aa1a1d2f237ef1026879d22b2d1660aa501d0d42cfe4c747" exitCode=0
Sep 16 13:33:32.171357 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.171302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerDied","Data":"d0e600a49e585ac5aa1a1d2f237ef1026879d22b2d1660aa501d0d42cfe4c747"}
Sep 16 13:33:32.172976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.172954    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mj7jz" event={"ID":"ccd06b3d-618d-4cab-950d-f86fe6bf7912","Type":"ContainerStarted","Data":"79e630d8f4d9677efbc08c0c5a7fa511d5bc4bd9c43f3684e60afd02b51b6f4b"}
Sep 16 13:33:32.174695 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.174672    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkg5d" event={"ID":"a32af615-2666-4782-807f-c7fa822b8706","Type":"ContainerStarted","Data":"d0213f271b2b10f0c4f5325e9ff1a33a4d2bb9811cb5074b8dffad70300f5c97"}
Sep 16 13:33:32.176477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.176452    2556 generic.go:356] "Generic (PLEG): container finished" podID="95e0be1d4166ccb5c95018175f7c6e1a" containerID="231ccf2d460e9d60a38c1ab840c778b6d3121350e1321f437168ca1b694a96fb" exitCode=0
Sep 16 13:33:32.176579 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.176535    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"95e0be1d4166ccb5c95018175f7c6e1a","Type":"ContainerDied","Data":"231ccf2d460e9d60a38c1ab840c778b6d3121350e1321f437168ca1b694a96fb"}
Sep 16 13:33:32.178278 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.178254    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52d77" event={"ID":"27850248-ce2d-480c-83f7-05c2477fb14f","Type":"ContainerStarted","Data":"0c888bf91c6cb521d37b573a4c7b371696fdc2071aa1cd089cae17bfe94570e7"}
Sep 16 13:33:32.179778 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.179735    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vp7zm" podStartSLOduration=6.553674996 podStartE2EDuration="22.179718699s" totalImagesPullingTime="15.626043703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:32.179176335 +0000 UTC m=+22.645113260" watchObservedRunningTime="2026-09-16 13:33:32.179718699 +0000 UTC m=+22.645655604"
Sep 16 13:33:32.179905 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.179887    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" event={"ID":"7dc59d1cdb7714ddbcf795e023ef2c6a","Type":"ContainerStarted","Data":"977acb084d41d575d8cd5f42024552c07a74d61ace74ce4ca57780616bb81bfb"}
Sep 16 13:33:32.191315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.191279    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rvzwh" podStartSLOduration=6.120569515 podStartE2EDuration="22.191265038s" totalImagesPullingTime="16.070695523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:32.190943483 +0000 UTC m=+22.656880381" watchObservedRunningTime="2026-09-16 13:33:32.191265038 +0000 UTC m=+22.657201945"
Sep 16 13:33:32.191660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.191644    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:32.192142 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.192126    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:32.205786 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.205734    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mj7jz" podStartSLOduration=2.152792787 podStartE2EDuration="22.205718712s" totalImagesPullingTime="20.052925925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:32.205118114 +0000 UTC m=+22.671055035" watchObservedRunningTime="2026-09-16 13:33:32.205718712 +0000 UTC m=+22.671655621"
Sep 16 13:33:32.217868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.217764    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-52d77" podStartSLOduration=1.990061544 podStartE2EDuration="22.21774967s" totalImagesPullingTime="20.227688126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:32.217485311 +0000 UTC m=+22.683422235" watchObservedRunningTime="2026-09-16 13:33:32.21774967 +0000 UTC m=+22.683686578"
Sep 16 13:33:32.248324 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.248271    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vkg5d" podStartSLOduration=2.079593747 podStartE2EDuration="22.248250708s" totalImagesPullingTime="20.168656961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:32.248213869 +0000 UTC m=+22.714150775" watchObservedRunningTime="2026-09-16 13:33:32.248250708 +0000 UTC m=+22.714187616"
Sep 16 13:33:32.294358 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.294311    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-251.ec2.internal" podStartSLOduration=21.29429531 podStartE2EDuration="21.29429531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:32.293774583 +0000 UTC m=+22.759711490" watchObservedRunningTime="2026-09-16 13:33:32.29429531 +0000 UTC m=+22.760232219"
Sep 16 13:33:32.393604 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.393560    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:32.446863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:32.446828    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:33.052257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.052229    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:33.052445 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:33.052359    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:33.130634 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.130610    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 13:33:33.184616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.184569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8qmv" event={"ID":"14ab078d-52b2-409b-b8e8-0e517825a6e7","Type":"ContainerStarted","Data":"94c0e69e816f5fb22f8a6eadbd63237f1c8b7417ba75700b6f94cd5ac1f5710d"}
Sep 16 13:33:33.186872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.186833    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" event={"ID":"95e0be1d4166ccb5c95018175f7c6e1a","Type":"ContainerStarted","Data":"71263e6977ba20abc0c53915902d6ac2dc68101ac84f0941045d23985021868e"}
Sep 16 13:33:33.189036 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.189013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt" event={"ID":"579cf366-9e9d-4bb1-b00f-2fb3270d5115","Type":"ContainerStarted","Data":"141c6262d856b03db873ef9b25f7273af341980cf2d788baedb78bfa6d0c376f"}
Sep 16 13:33:33.192390 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.192363    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"75040ba2161b67d2703364d4e00c04278c470b4e25432d91fc7db6525d2c27da"}
Sep 16 13:33:33.192507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.192425    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"592a4319b6ea6d722b35be5d51a01839c78e7c35b212c5ebb9b3526368712e80"}
Sep 16 13:33:33.193301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.193253    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:33.193543 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.193526    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rvzwh"
Sep 16 13:33:33.211752 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.211708    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k8qmv" podStartSLOduration=3.078219145 podStartE2EDuration="23.211692598s" totalImagesPullingTime="20.133473453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:33.199300721 +0000 UTC m=+23.665237629" watchObservedRunningTime="2026-09-16 13:33:33.211692598 +0000 UTC m=+23.677629507"
Sep 16 13:33:33.212274 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.212228    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-251.ec2.internal" podStartSLOduration=22.212216833 podStartE2EDuration="22.212216833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:33.211536132 +0000 UTC m=+23.677473058" watchObservedRunningTime="2026-09-16 13:33:33.212216833 +0000 UTC m=+23.678153743"
Sep 16 13:33:33.265267 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.265183    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:33.366582 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.366551    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:33.466838 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:33.466795    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:34.004601 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.004490    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T13:33:33.130628674Z","UUID":"75861786-930c-4bab-b957-05a94f0bebc8","Handler":null,"Name":"","Endpoint":""}
Sep 16 13:33:34.006872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.006850    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 13:33:34.006872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.006878    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 13:33:34.052498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.052465    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:34.052498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.052499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:34.052716 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:34.052613    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:34.053033 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:34.053001    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:34.500979 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.500733    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:34.601265 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:34.601156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:35.052376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:35.052341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:35.052558 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:35.052480    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:35.199110 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:35.199062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt" event={"ID":"579cf366-9e9d-4bb1-b00f-2fb3270d5115","Type":"ContainerStarted","Data":"b72f1997177b41ce33fe885dfd05ce7f195f7e739a627b3d6ae77e8b2489e23c"}
Sep 16 13:33:35.202255 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:35.202229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"3168c9de4024eb440c3ef2fac12eaec8a69856b60f14f0b981071c6ff72fafe9"}
Sep 16 13:33:35.218978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:35.218937    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872gt" podStartSLOduration=3.190119649 podStartE2EDuration="25.218922956s" totalImagesPullingTime="22.028803307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:35.217721268 +0000 UTC m=+25.683658167" watchObservedRunningTime="2026-09-16 13:33:35.218922956 +0000 UTC m=+25.684859864"
Sep 16 13:33:35.613950 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:35.613915    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:35.664108 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:35.664062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:36.052305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:36.052267    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:36.052537 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:36.052267    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:36.052537 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:36.052422    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:36.052537 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:36.052463    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:37.052910 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.052889    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:37.053268 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:37.052982    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:37.211238 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.211201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" event={"ID":"0ac88517-88f4-49f8-b15f-aa8d37c14c3f","Type":"ContainerStarted","Data":"e3f496593f5107fe208e17414c1d4a99cb0d762eb870fa0cc4d2dcd43c304519"}
Sep 16 13:33:37.211518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.211490    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:37.211642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.211558    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:37.227949 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.227925    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:37.231534 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.231512    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:37.239251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:37.237493    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp" podStartSLOduration=6.956676098 podStartE2EDuration="27.237473312s" totalImagesPullingTime="20.280797214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:37.236539289 +0000 UTC m=+27.702476205" watchObservedRunningTime="2026-09-16 13:33:37.237473312 +0000 UTC m=+27.703410220"
Sep 16 13:33:38.053476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.053278    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:38.054004 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.053276    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:38.054004 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:38.053564    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:38.054004 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:38.053641    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:38.215432 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.215397    2556 generic.go:356] "Generic (PLEG): container finished" podID="0c5a16d7-8099-4cf5-a759-1684dbbc46a0" containerID="069ccfd219935a027e724fa98851906131554ff37d3cad81eca108c64ad5c6f2" exitCode=0
Sep 16 13:33:38.215641 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.215481    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerDied","Data":"069ccfd219935a027e724fa98851906131554ff37d3cad81eca108c64ad5c6f2"}
Sep 16 13:33:38.215915 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.215896    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:33:38.408925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.408846    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:38.727653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:38.727620    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:39.053030 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.052993    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:39.053214 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:39.053150    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:39.224998 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.224972    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cxlj6"]
Sep 16 13:33:39.225626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.225069    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:39.225626 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:39.225180    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:39.229020 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.228040    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8w2s9"]
Sep 16 13:33:39.229020 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.228205    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:39.229020 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:39.228309    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:39.229020 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.228666    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9gskl"]
Sep 16 13:33:39.229020 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.228724    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:39.229020 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:39.228793    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:39.493742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:39.493712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:40.192762 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:40.192594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:40.221857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:40.221824    2556 generic.go:356] "Generic (PLEG): container finished" podID="0c5a16d7-8099-4cf5-a759-1684dbbc46a0" containerID="abd8d21e48458e9d32dd5a5d6c3c40c718a0f032eeeace529f0b6ac3c2f5ad13" exitCode=0
Sep 16 13:33:40.222021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:40.221868    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerDied","Data":"abd8d21e48458e9d32dd5a5d6c3c40c718a0f032eeeace529f0b6ac3c2f5ad13"}
Sep 16 13:33:40.244924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:40.244899    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:41.052661 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:41.052626    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:41.052838 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:41.052667    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:41.052838 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:41.052626    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:41.052838 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:41.052748    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:41.052838 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:41.052809    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:41.052993 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:41.052881    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:42.228830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:42.228800    2556 generic.go:356] "Generic (PLEG): container finished" podID="0c5a16d7-8099-4cf5-a759-1684dbbc46a0" containerID="c6f1541b4e383c0312907517b7c47eb32c4c61aa57c54067d8bf485f1c774d46" exitCode=0
Sep 16 13:33:42.229211 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:42.228852    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerDied","Data":"c6f1541b4e383c0312907517b7c47eb32c4c61aa57c54067d8bf485f1c774d46"}
Sep 16 13:33:42.703871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:42.703839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:42.703871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:42.703877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:42.704119 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.704082    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:42.704187 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.704171    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs podName:2786ffeb-964f-42fe-995f-e3be60ecd47f nodeName:}" failed. No retries permitted until 2026-09-16 13:34:14.704151329 +0000 UTC m=+65.170088221 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs") pod "network-metrics-daemon-8w2s9" (UID: "2786ffeb-964f-42fe-995f-e3be60ecd47f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 13:33:42.704580 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.704557    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:42.704721 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.704607    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret podName:54d8f078-4001-4193-8c85-2951ebc88278 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:14.704595395 +0000 UTC m=+65.170532280 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret") pod "global-pull-secret-syncer-cxlj6" (UID: "54d8f078-4001-4193-8c85-2951ebc88278") : object "kube-system"/"original-pull-secret" not registered
Sep 16 13:33:42.906333 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:42.906247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:42.906503 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.906398    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 13:33:42.906503 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.906418    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 13:33:42.906503 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.906431    2556 projected.go:196] Error preparing data for projected volume kube-api-access-rlmwl for pod openshift-network-diagnostics/network-check-target-9gskl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:42.906503 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:42.906496    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl podName:7dbd602d-d5b0-479e-8f26-ba2997c80f60 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:14.906477989 +0000 UTC m=+65.372414921 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rlmwl" (UniqueName: "kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl") pod "network-check-target-9gskl" (UID: "7dbd602d-d5b0-479e-8f26-ba2997c80f60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 13:33:43.052473 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:43.052447    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:43.052654 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:43.052539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:43.052654 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:43.052547    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gskl" podUID="7dbd602d-d5b0-479e-8f26-ba2997c80f60"
Sep 16 13:33:43.052654 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:43.052622    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:43.052778 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:43.052729    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w2s9" podUID="2786ffeb-964f-42fe-995f-e3be60ecd47f"
Sep 16 13:33:43.052778 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:43.052617    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cxlj6" podUID="54d8f078-4001-4193-8c85-2951ebc88278"
Sep 16 13:33:44.367716 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.367649    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-251.ec2.internal" event="NodeReady"
Sep 16 13:33:44.368057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.367781    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 13:33:44.410236 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.410203    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xcljx"]
Sep 16 13:33:44.412242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.412219    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.413215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.413196    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bj9zd"]
Sep 16 13:33:44.414573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.414423    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 13:33:44.414573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.414430    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 13:33:44.414573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.414532    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nnkqw\""
Sep 16 13:33:44.415018 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.415001    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.417266 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.417245    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2fr7x\""
Sep 16 13:33:44.417365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.417327    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 13:33:44.417465 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.417358    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 13:33:44.417687 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.417671    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 13:33:44.421611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.421593    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xcljx"]
Sep 16 13:33:44.425035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.425016    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bj9zd"]
Sep 16 13:33:44.517208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517183    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27eed164-faa1-4a17-ab02-a5f6528389bb-config-volume\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.517378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7jxq\" (UniqueName: \"kubernetes.io/projected/27eed164-faa1-4a17-ab02-a5f6528389bb-kube-api-access-x7jxq\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.517378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:44.517378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.517378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517257    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eed164-faa1-4a17-ab02-a5f6528389bb-tmp-dir\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.517378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517278    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa9b128f-810f-4a59-9ce9-c74bbe571128-cert\") pod \"ingress-canary-bj9zd\" (UID: \"fa9b128f-810f-4a59-9ce9-c74bbe571128\") " pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.517378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.517323    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmt2r\" (UniqueName: \"kubernetes.io/projected/fa9b128f-810f-4a59-9ce9-c74bbe571128-kube-api-access-hmt2r\") pod \"ingress-canary-bj9zd\" (UID: \"fa9b128f-810f-4a59-9ce9-c74bbe571128\") " pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.617712 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.617678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27eed164-faa1-4a17-ab02-a5f6528389bb-config-volume\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.617925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.617747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7jxq\" (UniqueName: \"kubernetes.io/projected/27eed164-faa1-4a17-ab02-a5f6528389bb-kube-api-access-x7jxq\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.617925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.617777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.617925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.617802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eed164-faa1-4a17-ab02-a5f6528389bb-tmp-dir\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.617925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.617843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa9b128f-810f-4a59-9ce9-c74bbe571128-cert\") pod \"ingress-canary-bj9zd\" (UID: \"fa9b128f-810f-4a59-9ce9-c74bbe571128\") " pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.617925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.617871    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmt2r\" (UniqueName: \"kubernetes.io/projected/fa9b128f-810f-4a59-9ce9-c74bbe571128-kube-api-access-hmt2r\") pod \"ingress-canary-bj9zd\" (UID: \"fa9b128f-810f-4a59-9ce9-c74bbe571128\") " pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.618232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.618016    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:44.618232 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:44.618167    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:33:44.618232 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:44.618223    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls podName:27eed164-faa1-4a17-ab02-a5f6528389bb nodeName:}" failed. No retries permitted until 2026-09-16 13:33:45.118205938 +0000 UTC m=+35.584142827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls") pod "dns-default-xcljx" (UID: "27eed164-faa1-4a17-ab02-a5f6528389bb") : secret "dns-default-metrics-tls" not found
Sep 16 13:33:44.618381 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.618335    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eed164-faa1-4a17-ab02-a5f6528389bb-tmp-dir\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.618485 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.618461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/27eed164-faa1-4a17-ab02-a5f6528389bb-config-volume\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.623310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.623282    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fa9b128f-810f-4a59-9ce9-c74bbe571128-cert\") pod \"ingress-canary-bj9zd\" (UID: \"fa9b128f-810f-4a59-9ce9-c74bbe571128\") " pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.627589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.627286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7jxq\" (UniqueName: \"kubernetes.io/projected/27eed164-faa1-4a17-ab02-a5f6528389bb-kube-api-access-x7jxq\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:44.627705 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.627629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmt2r\" (UniqueName: \"kubernetes.io/projected/fa9b128f-810f-4a59-9ce9-c74bbe571128-kube-api-access-hmt2r\") pod \"ingress-canary-bj9zd\" (UID: \"fa9b128f-810f-4a59-9ce9-c74bbe571128\") " pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:44.732670 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:44.732631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bj9zd"
Sep 16 13:33:45.019525 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.019293    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bj9zd"]
Sep 16 13:33:45.052983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.052956    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:33:45.053134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.052999    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:33:45.053134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.053065    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:33:45.055731 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.055702    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2cz7s\""
Sep 16 13:33:45.055842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.055791    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 13:33:45.055842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.055796    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 13:33:45.055842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.055811    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 13:33:45.056233 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.056206    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69tl6\""
Sep 16 13:33:45.056325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.056247    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 13:33:45.121825 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.121789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:45.121970 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:45.121941    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:33:45.122010 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:45.122003    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls podName:27eed164-faa1-4a17-ab02-a5f6528389bb nodeName:}" failed. No retries permitted until 2026-09-16 13:33:46.121987019 +0000 UTC m=+36.587923904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls") pod "dns-default-xcljx" (UID: "27eed164-faa1-4a17-ab02-a5f6528389bb") : secret "dns-default-metrics-tls" not found
Sep 16 13:33:45.237571 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:45.237533    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bj9zd" event={"ID":"fa9b128f-810f-4a59-9ce9-c74bbe571128","Type":"ContainerStarted","Data":"76b71dd2be2315ca759ca8c58383f7668af67b683a51ec8028b9193b13e5e791"}
Sep 16 13:33:46.128392 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:46.128331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:46.128898 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:46.128465    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:33:46.128898 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:46.128549    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls podName:27eed164-faa1-4a17-ab02-a5f6528389bb nodeName:}" failed. No retries permitted until 2026-09-16 13:33:48.128528023 +0000 UTC m=+38.594464925 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls") pod "dns-default-xcljx" (UID: "27eed164-faa1-4a17-ab02-a5f6528389bb") : secret "dns-default-metrics-tls" not found
Sep 16 13:33:47.128462 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:47.128418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:47.178920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:47.178892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:47.245551 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:47.245477    2556 generic.go:356] "Generic (PLEG): container finished" podID="0c5a16d7-8099-4cf5-a759-1684dbbc46a0" containerID="3167cabead6b0b4e944d43d9eb17601000216b8c7121a2828167c2e1292ed154" exitCode=0
Sep 16 13:33:47.245551 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:47.245531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerDied","Data":"3167cabead6b0b4e944d43d9eb17601000216b8c7121a2828167c2e1292ed154"}
Sep 16 13:33:48.143199 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:48.143162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:48.143650 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:48.143273    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:33:48.143650 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:48.143325    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls podName:27eed164-faa1-4a17-ab02-a5f6528389bb nodeName:}" failed. No retries permitted until 2026-09-16 13:33:52.143311845 +0000 UTC m=+42.609248731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls") pod "dns-default-xcljx" (UID: "27eed164-faa1-4a17-ab02-a5f6528389bb") : secret "dns-default-metrics-tls" not found
Sep 16 13:33:48.251464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:48.251432    2556 generic.go:356] "Generic (PLEG): container finished" podID="0c5a16d7-8099-4cf5-a759-1684dbbc46a0" containerID="93d7acff735ef248b7cec7d51d2e1c3c270850649bd8b9caa0c36d572ea8c46f" exitCode=0
Sep 16 13:33:48.251644 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:48.251473    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerDied","Data":"93d7acff735ef248b7cec7d51d2e1c3c270850649bd8b9caa0c36d572ea8c46f"}
Sep 16 13:33:49.257272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:49.257229    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-x92sh" event={"ID":"0c5a16d7-8099-4cf5-a759-1684dbbc46a0","Type":"ContainerStarted","Data":"2b8bb73958e2b0a41d9dfc6861fab2e114107b3c338add534de355777bf5abe7"}
Sep 16 13:33:49.276667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:49.276621    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-x92sh" podStartSLOduration=7.42881338 podStartE2EDuration="39.276604705s" totalImagesPullingTime="31.847791325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:49.275566195 +0000 UTC m=+39.741503101" watchObservedRunningTime="2026-09-16 13:33:49.276604705 +0000 UTC m=+39.742541611"
Sep 16 13:33:49.405174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:49.405123    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:49.464785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:49.464544    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:49.526996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:49.526971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:50.261466 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:50.261430    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bj9zd" event={"ID":"fa9b128f-810f-4a59-9ce9-c74bbe571128","Type":"ContainerStarted","Data":"4902656a97213f18a02f03b68a3d3c6c4208e2455845316cb8e5faec6f240d2e"}
Sep 16 13:33:50.276722 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:50.276667    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bj9zd" podStartSLOduration=1.880827517 podStartE2EDuration="6.276650601s" totalImagesPullingTime="4.395823084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:50.275919916 +0000 UTC m=+40.741856849" watchObservedRunningTime="2026-09-16 13:33:50.276650601 +0000 UTC m=+40.742587536"
Sep 16 13:33:50.583698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:50.583614    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:50.642342 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:50.642303    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:51.738795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:51.738764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:51.839837 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:51.839810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:52.173539 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:52.173495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:33:52.173732 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:52.173638    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:33:52.173732 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:52.173713    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls podName:27eed164-faa1-4a17-ab02-a5f6528389bb nodeName:}" failed. No retries permitted until 2026-09-16 13:34:00.173694129 +0000 UTC m=+50.639631028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls") pod "dns-default-xcljx" (UID: "27eed164-faa1-4a17-ab02-a5f6528389bb") : secret "dns-default-metrics-tls" not found
Sep 16 13:33:53.134225 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.134170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:53.188229 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.188201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:53.382660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.382629    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"]
Sep 16 13:33:53.404453 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.404385    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"]
Sep 16 13:33:53.404604 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.404493    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.407893 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.407873    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 13:33:53.407893 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.407872    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 13:33:53.408820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.408801    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 13:33:53.408923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.408831    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vnpgd\""
Sep 16 13:33:53.408923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.408834    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 13:33:53.485793 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.485764    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"]
Sep 16 13:33:53.502369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.502331    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b79cff5dd-mnnh2"]
Sep 16 13:33:53.502566 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.502541    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.504875 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.504662    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 13:33:53.505185 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.505158    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:33:53.505533 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.505512    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 13:33:53.505725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.505707    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4gjsc\""
Sep 16 13:33:53.505863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.505763    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 13:33:53.528521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.528499    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"]
Sep 16 13:33:53.528616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.528529    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b79cff5dd-mnnh2"]
Sep 16 13:33:53.528616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.528606    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.530745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.530723    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 13:33:53.530898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.530879    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xldgt\""
Sep 16 13:33:53.530954 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.530936    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 13:33:53.531015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.530963    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 13:33:53.531015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.530973    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 13:33:53.531015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.530985    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 13:33:53.531180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.531070    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 13:33:53.583863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.583828    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"]
Sep 16 13:33:53.585528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.585502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.585681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.585543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb6v6\" (UniqueName: \"kubernetes.io/projected/68537ef5-22fb-4534-9627-5441ca8ac4a5-kube-api-access-vb6v6\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.585681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.585590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/68537ef5-22fb-4534-9627-5441ca8ac4a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.608330 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.608300    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"]
Sep 16 13:33:53.608464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.608384    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.610596 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.610577    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 13:33:53.610596 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.610600    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-htmqr\""
Sep 16 13:33:53.610758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.610653    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 13:33:53.610872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.610857    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 13:33:53.611482 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.611466    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:33:53.634190 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.634166    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:53.683657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.683633    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:53.686726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.686786 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.686786 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686766    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.686868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686784    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb6v6\" (UniqueName: \"kubernetes.io/projected/68537ef5-22fb-4534-9627-5441ca8ac4a5-kube-api-access-vb6v6\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.686868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686802    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-286gd\" (UniqueName: \"kubernetes.io/projected/382a1e6d-8b4a-4948-82b0-223d7a8d9564-kube-api-access-286gd\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.686868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-stats-auth\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.686868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686840    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/68537ef5-22fb-4534-9627-5441ca8ac4a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.686868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/382a1e6d-8b4a-4948-82b0-223d7a8d9564-config\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.687109 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:53.686868    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:53.687109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-default-certificate\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.687109 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:53.686931    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls podName:68537ef5-22fb-4534-9627-5441ca8ac4a5 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:54.186916238 +0000 UTC m=+44.652853128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bzw76" (UID: "68537ef5-22fb-4534-9627-5441ca8ac4a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:53.687109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.686978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsh2h\" (UniqueName: \"kubernetes.io/projected/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-kube-api-access-xsh2h\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.687109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.687014    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/382a1e6d-8b4a-4948-82b0-223d7a8d9564-serving-cert\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.687536 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.687520    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/68537ef5-22fb-4534-9627-5441ca8ac4a5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.695489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.695473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb6v6\" (UniqueName: \"kubernetes.io/projected/68537ef5-22fb-4534-9627-5441ca8ac4a5-kube-api-access-vb6v6\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:53.787921 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.787888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-default-certificate\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.787921 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.787921    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsh2h\" (UniqueName: \"kubernetes.io/projected/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-kube-api-access-xsh2h\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.787945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtt8n\" (UniqueName: \"kubernetes.io/projected/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-kube-api-access-jtt8n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.787963    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/382a1e6d-8b4a-4948-82b0-223d7a8d9564-serving-cert\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.787989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:53.788147    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:33:54.288123015 +0000 UTC m=+44.754059903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : configmap references non-existent config key: service-ca.crt
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:53.788180    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 13:33:53.788198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788188    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.788521 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:53.788218    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:33:54.288207897 +0000 UTC m=+44.754144784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : secret "router-metrics-certs-default" not found
Sep 16 13:33:53.788521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-286gd\" (UniqueName: \"kubernetes.io/projected/382a1e6d-8b4a-4948-82b0-223d7a8d9564-kube-api-access-286gd\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.788521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788284    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-stats-auth\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.788521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788310    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/382a1e6d-8b4a-4948-82b0-223d7a8d9564-config\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.788821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.788800    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/382a1e6d-8b4a-4948-82b0-223d7a8d9564-config\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.790565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.790535    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/382a1e6d-8b4a-4948-82b0-223d7a8d9564-serving-cert\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.790669 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.790652    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-default-certificate\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.797987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.797962    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-286gd\" (UniqueName: \"kubernetes.io/projected/382a1e6d-8b4a-4948-82b0-223d7a8d9564-kube-api-access-286gd\") pod \"service-ca-operator-cf7c776b7-kg625\" (UID: \"382a1e6d-8b4a-4948-82b0-223d7a8d9564\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.802015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.801996    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-stats-auth\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.803783 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.803767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsh2h\" (UniqueName: \"kubernetes.io/projected/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-kube-api-access-xsh2h\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:53.813622 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.813604    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"
Sep 16 13:33:53.889658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.889628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.889777 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.889696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.889837 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.889782    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtt8n\" (UniqueName: \"kubernetes.io/projected/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-kube-api-access-jtt8n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.890338 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.890313    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.892421 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.892397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.897618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.897597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtt8n\" (UniqueName: \"kubernetes.io/projected/d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3-kube-api-access-jtt8n\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b4lwr\" (UID: \"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:53.916662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:53.916638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"
Sep 16 13:33:54.094067 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.094033    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625"]
Sep 16 13:33:54.096865 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:54.096840    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382a1e6d_8b4a_4948_82b0_223d7a8d9564.slice/crio-9dc1579cbee40297391dcdda0f7329c4629e005901e61bea122083abd5172918 WatchSource:0}: Error finding container 9dc1579cbee40297391dcdda0f7329c4629e005901e61bea122083abd5172918: Status 404 returned error can't find the container with id 9dc1579cbee40297391dcdda0f7329c4629e005901e61bea122083abd5172918
Sep 16 13:33:54.192638 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.192561    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:54.193065 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:54.192755    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:54.193065 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:54.192839    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls podName:68537ef5-22fb-4534-9627-5441ca8ac4a5 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:55.19281762 +0000 UTC m=+45.658754510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bzw76" (UID: "68537ef5-22fb-4534-9627-5441ca8ac4a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:54.197238 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.197186    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr"]
Sep 16 13:33:54.201161 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:54.201133    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20ab845_bbe8_4d3f_ad4e_f22f6fd062f3.slice/crio-2010cd99ca9a4c2bc0d1a6694f97251615bdec42c2a5db26137781f3751f8397 WatchSource:0}: Error finding container 2010cd99ca9a4c2bc0d1a6694f97251615bdec42c2a5db26137781f3751f8397: Status 404 returned error can't find the container with id 2010cd99ca9a4c2bc0d1a6694f97251615bdec42c2a5db26137781f3751f8397
Sep 16 13:33:54.271680 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.271646    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr" event={"ID":"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3","Type":"ContainerStarted","Data":"2010cd99ca9a4c2bc0d1a6694f97251615bdec42c2a5db26137781f3751f8397"}
Sep 16 13:33:54.272856 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.272831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625" event={"ID":"382a1e6d-8b4a-4948-82b0-223d7a8d9564","Type":"ContainerStarted","Data":"9dc1579cbee40297391dcdda0f7329c4629e005901e61bea122083abd5172918"}
Sep 16 13:33:54.293213 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.293190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:54.293304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:54.293235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:54.293362 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:54.293348    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:33:55.293328218 +0000 UTC m=+45.759265104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : configmap references non-existent config key: service-ca.crt
Sep 16 13:33:54.293362 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:54.293354    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 13:33:54.293458 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:54.293438    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:33:55.293405134 +0000 UTC m=+45.759342022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : secret "router-metrics-certs-default" not found
Sep 16 13:33:55.200459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:55.200418    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:55.200891 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:55.200569    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:55.200891 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:55.200638    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls podName:68537ef5-22fb-4534-9627-5441ca8ac4a5 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:57.200619697 +0000 UTC m=+47.666556601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bzw76" (UID: "68537ef5-22fb-4534-9627-5441ca8ac4a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:55.301489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:55.301450    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:55.301665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:55.301513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:55.301665 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:55.301634    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 13:33:55.301665 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:55.301644    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:33:57.301625748 +0000 UTC m=+47.767562651 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : configmap references non-existent config key: service-ca.crt
Sep 16 13:33:55.301806 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:55.301670    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:33:57.301659379 +0000 UTC m=+47.767596265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : secret "router-metrics-certs-default" not found
Sep 16 13:33:57.218928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:57.218890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:33:57.219420 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:57.219066    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:57.219420 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:57.219163    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls podName:68537ef5-22fb-4534-9627-5441ca8ac4a5 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:01.219141583 +0000 UTC m=+51.685078486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bzw76" (UID: "68537ef5-22fb-4534-9627-5441ca8ac4a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 13:33:57.320136 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:57.320081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:57.320302 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:57.320212    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 13:33:57.320302 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:57.320263    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:33:57.320302 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:57.320286    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:34:01.32026481 +0000 UTC m=+51.786201720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : secret "router-metrics-certs-default" not found
Sep 16 13:33:57.320475 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:57.320373    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:34:01.320357138 +0000 UTC m=+51.786294025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : configmap references non-existent config key: service-ca.crt
Sep 16 13:33:57.569266 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:57.569193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:57.630871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:57.630837    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:58.284617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.284574    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625" event={"ID":"382a1e6d-8b4a-4948-82b0-223d7a8d9564","Type":"ContainerStarted","Data":"5ed9266f7f4c206951fd1bec0b3a585c10f0f45f51965a67e52124975c05fe8d"}
Sep 16 13:33:58.286236 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.286211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr" event={"ID":"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3","Type":"ContainerStarted","Data":"8688d17f9146c5ed2fa37b5a44e19109840ae504e55b63a205e9cf7cfe1f6653"}
Sep 16 13:33:58.299632 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.299576    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625" podStartSLOduration=1.528120532 podStartE2EDuration="5.299557974s" totalImagesPullingTime="3.771437442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:58.298720422 +0000 UTC m=+48.764657341" watchObservedRunningTime="2026-09-16 13:33:58.299557974 +0000 UTC m=+48.765494896"
Sep 16 13:33:58.312037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.311982    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr" podStartSLOduration=1.638995379 podStartE2EDuration="5.311963374s" totalImagesPullingTime="3.672967995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:33:58.311180663 +0000 UTC m=+48.777117589" watchObservedRunningTime="2026-09-16 13:33:58.311963374 +0000 UTC m=+48.777900283"
Sep 16 13:33:58.421900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.421862    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"]
Sep 16 13:33:58.442686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.442661    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"]
Sep 16 13:33:58.442828 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.442751    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"
Sep 16 13:33:58.452509 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.452485    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qphl4\""
Sep 16 13:33:58.531369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.531328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smmc4\" (UniqueName: \"kubernetes.io/projected/05a8495a-32d3-411a-873e-1671e4939b36-kube-api-access-smmc4\") pod \"network-check-source-6f8b768455-96fs4\" (UID: \"05a8495a-32d3-411a-873e-1671e4939b36\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"
Sep 16 13:33:58.632311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.632211    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smmc4\" (UniqueName: \"kubernetes.io/projected/05a8495a-32d3-411a-873e-1671e4939b36-kube-api-access-smmc4\") pod \"network-check-source-6f8b768455-96fs4\" (UID: \"05a8495a-32d3-411a-873e-1671e4939b36\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"
Sep 16 13:33:58.660130 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.660080    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smmc4\" (UniqueName: \"kubernetes.io/projected/05a8495a-32d3-411a-873e-1671e4939b36-kube-api-access-smmc4\") pod \"network-check-source-6f8b768455-96fs4\" (UID: \"05a8495a-32d3-411a-873e-1671e4939b36\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"
Sep 16 13:33:58.701236 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.701194    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:58.714187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.714158    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qcnrn"]
Sep 16 13:33:58.733797 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.733763    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qcnrn"]
Sep 16 13:33:58.733950 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.733900    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.735997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.735974    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 13:33:58.736308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.736290    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sjsnp\""
Sep 16 13:33:58.736383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.736317    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 13:33:58.736684 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.736666    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 13:33:58.736787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.736690    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 13:33:58.753742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.753721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"
Sep 16 13:33:58.778682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.778653    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:58.833367 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.833339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-crio-socket\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.833531 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.833370    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.833531 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.833412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6dk\" (UniqueName: \"kubernetes.io/projected/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-api-access-lq6dk\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.833531 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.833459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.833531 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.833492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-data-volume\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.934687 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.934644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.934873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.934725    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-data-volume\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.934873 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:58.934761    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:33:58.934873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.934787    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-crio-socket\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.934873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.934812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.934873 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:58.934827    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:59.434810559 +0000 UTC m=+49.900747459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : configmap "kube-rbac-proxy" not found
Sep 16 13:33:58.935123 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.934888    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6dk\" (UniqueName: \"kubernetes.io/projected/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-api-access-lq6dk\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.935123 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:58.934917    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:33:58.935123 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:58.934968    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:33:59.434951915 +0000 UTC m=+49.900888820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:33:58.935123 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.935024    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-crio-socket\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.935123 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.935043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-data-volume\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:58.943263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:58.943238    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6dk\" (UniqueName: \"kubernetes.io/projected/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-api-access-lq6dk\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:59.033756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.033720    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-96fs4"]
Sep 16 13:33:59.036896 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:33:59.036864    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a8495a_32d3_411a_873e_1671e4939b36.slice/crio-fa147760f5d62515a6324483f927c2c536221cf1cd0e02527b5c3e148393fa21 WatchSource:0}: Error finding container fa147760f5d62515a6324483f927c2c536221cf1cd0e02527b5c3e148393fa21: Status 404 returned error can't find the container with id fa147760f5d62515a6324483f927c2c536221cf1cd0e02527b5c3e148393fa21
Sep 16 13:33:59.290474 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.290388    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4" event={"ID":"05a8495a-32d3-411a-873e-1671e4939b36","Type":"ContainerStarted","Data":"fa147760f5d62515a6324483f927c2c536221cf1cd0e02527b5c3e148393fa21"}
Sep 16 13:33:59.438483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.438454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:59.438658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.438523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:33:59.438658 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:59.438574    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:33:59.438658 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:59.438621    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:33:59.438658 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:59.438638    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:00.438622871 +0000 UTC m=+50.904559757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : configmap "kube-rbac-proxy" not found
Sep 16 13:33:59.438658 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:33:59.438658    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:00.438647311 +0000 UTC m=+50.904584197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:33:59.652150 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.652038    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:59.750938 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.750910    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:33:59.851343 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:33:59.851303    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:00.244188 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:00.244137    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:34:00.244391 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:00.244320    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 13:34:00.244464 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:00.244394    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls podName:27eed164-faa1-4a17-ab02-a5f6528389bb nodeName:}" failed. No retries permitted until 2026-09-16 13:34:16.244372799 +0000 UTC m=+66.710309687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls") pod "dns-default-xcljx" (UID: "27eed164-faa1-4a17-ab02-a5f6528389bb") : secret "dns-default-metrics-tls" not found
Sep 16 13:34:00.446564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:00.446525    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:00.447045 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:00.446620    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:00.447045 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:00.446662    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:34:00.447045 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:00.446739    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:02.446720732 +0000 UTC m=+52.912657638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : configmap "kube-rbac-proxy" not found
Sep 16 13:34:00.447045 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:00.446744    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:34:00.447045 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:00.446800    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:02.446786775 +0000 UTC m=+52.912723661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:34:00.456007 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:00.455985    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52d77_27850248-ce2d-480c-83f7-05c2477fb14f/dns-node-resolver/0.log"
Sep 16 13:34:00.705408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:00.705368    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:00.763940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:00.763894    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:01.252567 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.252528    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:34:01.252751 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:01.252702    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 13:34:01.252805 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:01.252772    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls podName:68537ef5-22fb-4534-9627-5441ca8ac4a5 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:09.252754043 +0000 UTC m=+59.718690930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bzw76" (UID: "68537ef5-22fb-4534-9627-5441ca8ac4a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 13:34:01.298070 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.298042    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dlqs9"]
Sep 16 13:34:01.314175 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.314139    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dlqs9"]
Sep 16 13:34:01.314337 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.314192    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.316591 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.316562    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 13:34:01.316715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.316564    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 13:34:01.317278 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.317249    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 13:34:01.317398 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.317360    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-grhsj\""
Sep 16 13:34:01.317398 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.317390    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 13:34:01.337258 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.337234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:01.353443 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.353418    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-signing-key\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.353584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.353449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2t7r\" (UniqueName: \"kubernetes.io/projected/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-kube-api-access-l2t7r\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.353584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.353518    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:01.353584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.353547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:01.353742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.353602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-signing-cabundle\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.353742 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:01.353656    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 13:34:01.353742 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:01.353694    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:34:09.353670525 +0000 UTC m=+59.819607432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : configmap references non-existent config key: service-ca.crt
Sep 16 13:34:01.353742 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:01.353721    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:34:09.353709482 +0000 UTC m=+59.819646367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : secret "router-metrics-certs-default" not found
Sep 16 13:34:01.454774 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.454715    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-signing-cabundle\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.455261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.454903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-signing-key\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.455261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.454939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2t7r\" (UniqueName: \"kubernetes.io/projected/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-kube-api-access-l2t7r\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.455476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.455456    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-signing-cabundle\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.458006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.457981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-signing-key\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.463757 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.463732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2t7r\" (UniqueName: \"kubernetes.io/projected/eeb6a25b-151b-41a4-bf33-a9e82ace7e79-kube-api-access-l2t7r\") pod \"service-ca-568869fdc5-dlqs9\" (UID: \"eeb6a25b-151b-41a4-bf33-a9e82ace7e79\") " pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.624769 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.624680    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-dlqs9"
Sep 16 13:34:01.655149 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.655119    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vp7zm_da85859e-800e-4d7c-8b95-775d36698fdd/node-ca/0.log"
Sep 16 13:34:01.682773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:01.682734    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:02.255516 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:02.255482    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bj9zd_fa9b128f-810f-4a59-9ce9-c74bbe571128/serve-healthcheck-canary/0.log"
Sep 16 13:34:02.464590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:02.464549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:02.464966 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:02.464614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:02.464966 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:02.464678    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:34:02.464966 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:02.464741    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:06.464725956 +0000 UTC m=+56.930662843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : configmap "kube-rbac-proxy" not found
Sep 16 13:34:02.464966 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:02.464780    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:34:02.464966 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:02.464836    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:06.464822503 +0000 UTC m=+56.930759390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:34:02.491452 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:02.491376    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-dlqs9"]
Sep 16 13:34:02.574383 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:02.574345    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb6a25b_151b_41a4_bf33_a9e82ace7e79.slice/crio-323f4649e63285935a10d8bca0150afbf836408336ff93efe1455b20e0850a76 WatchSource:0}: Error finding container 323f4649e63285935a10d8bca0150afbf836408336ff93efe1455b20e0850a76: Status 404 returned error can't find the container with id 323f4649e63285935a10d8bca0150afbf836408336ff93efe1455b20e0850a76
Sep 16 13:34:03.165525 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.165483    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:03.303147 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.303110    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-dlqs9" event={"ID":"eeb6a25b-151b-41a4-bf33-a9e82ace7e79","Type":"ContainerStarted","Data":"d7af0b418b11d2f2356dd4a5870f0ae6cc8e818156b1c91017bc4960389965ea"}
Sep 16 13:34:03.303350 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.303211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-dlqs9" event={"ID":"eeb6a25b-151b-41a4-bf33-a9e82ace7e79","Type":"ContainerStarted","Data":"323f4649e63285935a10d8bca0150afbf836408336ff93efe1455b20e0850a76"}
Sep 16 13:34:03.304658 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.304630    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4" event={"ID":"05a8495a-32d3-411a-873e-1671e4939b36","Type":"ContainerStarted","Data":"b6df0f550d5bf9402fb3f8e15a9cd83ba2ce41c0dfa8ca924470aed2238f60d1"}
Sep 16 13:34:03.321264 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.321218    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-dlqs9" podStartSLOduration=2.321202716 podStartE2EDuration="2.321202716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:03.320057495 +0000 UTC m=+53.785994403" watchObservedRunningTime="2026-09-16 13:34:03.321202716 +0000 UTC m=+53.787139624"
Sep 16 13:34:03.336277 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.336225    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-96fs4" podStartSLOduration=1.761894264 podStartE2EDuration="5.336205113s" totalImagesPullingTime="3.574310849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:03.335616765 +0000 UTC m=+53.801553673" watchObservedRunningTime="2026-09-16 13:34:03.336205113 +0000 UTC m=+53.802142023"
Sep 16 13:34:03.827588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.827507    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:03.936706 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:03.936671    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:04.248494 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:04.248457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:04.347911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:04.347864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:04.448728 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:04.448696    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:04.549134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:04.549026    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:05.562830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:05.562803    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:05.622973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:05.622923    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:05.927555 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:05.927525    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:06.126771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:06.126742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:06.328541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:06.328455    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:06.498626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:06.498583    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:06.498806 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:06.498646    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:06.498806 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:06.498707    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 13:34:06.498806 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:06.498751    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 13:34:06.498806 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:06.498770    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:14.498754198 +0000 UTC m=+64.964691083 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : configmap "kube-rbac-proxy" not found
Sep 16 13:34:06.498806 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:06.498791    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls podName:e9af322c-e51f-4ca3-ae75-ac4dc4e7da37 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:14.498779362 +0000 UTC m=+64.964716249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qcnrn" (UID: "e9af322c-e51f-4ca3-ae75-ac4dc4e7da37") : secret "insights-runtime-extractor-tls" not found
Sep 16 13:34:06.521551 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:06.521517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:06.730250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:06.730214    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:07.608696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:07.608663    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:07.717355 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:07.717313    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:07.818694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:07.818666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:07.922228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:07.922191    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:08.127324 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:08.127285    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:08.322471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:08.322388    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:08.372287 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:08.372253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:08.524715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:08.524678    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:08.956211 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:08.956177    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:09.240332 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.240247    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2j9wp"
Sep 16 13:34:09.321573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.321533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:34:09.321767 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:09.321663    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 13:34:09.321767 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:09.321732    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls podName:68537ef5-22fb-4534-9627-5441ca8ac4a5 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:25.321710797 +0000 UTC m=+75.787647685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-bzw76" (UID: "68537ef5-22fb-4534-9627-5441ca8ac4a5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 13:34:09.422562 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.422522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:09.422562 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.422569    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:09.422769 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:09.422687    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle podName:8d999b14-0ce5-49e5-8c6e-3c74fedd210c nodeName:}" failed. No retries permitted until 2026-09-16 13:34:25.422668344 +0000 UTC m=+75.888605230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle") pod "router-default-5b79cff5dd-mnnh2" (UID: "8d999b14-0ce5-49e5-8c6e-3c74fedd210c") : configmap references non-existent config key: service-ca.crt
Sep 16 13:34:09.425126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.425079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-metrics-certs\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:09.524571 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.524446    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:09.723575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:09.723541    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:10.123850 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:10.123788    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:10.224873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:10.224836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:10.324688 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:10.324647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:10.919791 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:10.919731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:11.328990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:11.328911    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:11.722301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:11.722267    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:11.828222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:11.828190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:12.123732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:12.123656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:12.322311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:12.322275    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:12.720654 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:12.720613    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:13.322292 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:13.322257    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:13.720874 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:13.720840    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:14.117647 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.117565    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:14.329513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.329478    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:14.563633 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.563601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:14.563810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.563649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:14.564173 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.564154    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:14.566029 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.566010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e9af322c-e51f-4ca3-ae75-ac4dc4e7da37-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qcnrn\" (UID: \"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37\") " pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:14.646775 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.646751    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sjsnp\""
Sep 16 13:34:14.655340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.655320    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qcnrn"
Sep 16 13:34:14.766725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.766689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:34:14.767132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.766728    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:34:14.768937 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.768920    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 13:34:14.768993 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.768920    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 13:34:14.779777 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.779756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/54d8f078-4001-4193-8c85-2951ebc88278-original-pull-secret\") pod \"global-pull-secret-syncer-cxlj6\" (UID: \"54d8f078-4001-4193-8c85-2951ebc88278\") " pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:34:14.779885 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.779761    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2786ffeb-964f-42fe-995f-e3be60ecd47f-metrics-certs\") pod \"network-metrics-daemon-8w2s9\" (UID: \"2786ffeb-964f-42fe-995f-e3be60ecd47f\") " pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:34:14.929726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.929672    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qcnrn"]
Sep 16 13:34:14.932616 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:14.932590    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9af322c_e51f_4ca3_ae75_ac4dc4e7da37.slice/crio-7121bb21e8a4bc59dc27d2641655127be4b0237a80b1b0034bd02e1b705d22cb WatchSource:0}: Error finding container 7121bb21e8a4bc59dc27d2641655127be4b0237a80b1b0034bd02e1b705d22cb: Status 404 returned error can't find the container with id 7121bb21e8a4bc59dc27d2641655127be4b0237a80b1b0034bd02e1b705d22cb
Sep 16 13:34:14.967831 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.967805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:34:14.970352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:14.970333    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmwl\" (UniqueName: \"kubernetes.io/projected/7dbd602d-d5b0-479e-8f26-ba2997c80f60-kube-api-access-rlmwl\") pod \"network-check-target-9gskl\" (UID: \"7dbd602d-d5b0-479e-8f26-ba2997c80f60\") " pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:34:15.067399 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.067368    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-69tl6\""
Sep 16 13:34:15.073641 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.073621    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2cz7s\""
Sep 16 13:34:15.076017 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.076000    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:34:15.077694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.077676    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxlj6"
Sep 16 13:34:15.082047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.082024    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w2s9"
Sep 16 13:34:15.290359 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.290280    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:15.345797 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.345755    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qcnrn" event={"ID":"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37","Type":"ContainerStarted","Data":"7d683361c184c38e00fdbe4414ba2b93808abd385bbd589671bd8f675d58e378"}
Sep 16 13:34:15.345925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.345864    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qcnrn" event={"ID":"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37","Type":"ContainerStarted","Data":"7121bb21e8a4bc59dc27d2641655127be4b0237a80b1b0034bd02e1b705d22cb"}
Sep 16 13:34:15.355649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.355508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9gskl"]
Sep 16 13:34:15.358628 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:15.358596    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbd602d_d5b0_479e_8f26_ba2997c80f60.slice/crio-03f099e6ad4baa90b1c941d550de7090e4d8c1fdf50bd03de129eaf480a1d51c WatchSource:0}: Error finding container 03f099e6ad4baa90b1c941d550de7090e4d8c1fdf50bd03de129eaf480a1d51c: Status 404 returned error can't find the container with id 03f099e6ad4baa90b1c941d550de7090e4d8c1fdf50bd03de129eaf480a1d51c
Sep 16 13:34:15.358769 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.358676    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cxlj6"]
Sep 16 13:34:15.361283 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:15.360658    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8w2s9"]
Sep 16 13:34:15.361912 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:15.361884    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d8f078_4001_4193_8c85_2951ebc88278.slice/crio-6fcc5aeb23d910ebbb2657ef05c39f903d6aa12b229e4bc083cc1f9b65719590 WatchSource:0}: Error finding container 6fcc5aeb23d910ebbb2657ef05c39f903d6aa12b229e4bc083cc1f9b65719590: Status 404 returned error can't find the container with id 6fcc5aeb23d910ebbb2657ef05c39f903d6aa12b229e4bc083cc1f9b65719590
Sep 16 13:34:15.363741 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:15.363718    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2786ffeb_964f_42fe_995f_e3be60ecd47f.slice/crio-8635cf77541f5cf3bf1f5c0fd771e3451a3ca64c44f78436fffa811fbe8876bb WatchSource:0}: Error finding container 8635cf77541f5cf3bf1f5c0fd771e3451a3ca64c44f78436fffa811fbe8876bb: Status 404 returned error can't find the container with id 8635cf77541f5cf3bf1f5c0fd771e3451a3ca64c44f78436fffa811fbe8876bb
Sep 16 13:34:16.278674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.278641    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:34:16.281334 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.281304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/27eed164-faa1-4a17-ab02-a5f6528389bb-metrics-tls\") pod \"dns-default-xcljx\" (UID: \"27eed164-faa1-4a17-ab02-a5f6528389bb\") " pod="openshift-dns/dns-default-xcljx"
Sep 16 13:34:16.351330 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.351245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w2s9" event={"ID":"2786ffeb-964f-42fe-995f-e3be60ecd47f","Type":"ContainerStarted","Data":"8635cf77541f5cf3bf1f5c0fd771e3451a3ca64c44f78436fffa811fbe8876bb"}
Sep 16 13:34:16.353920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.353844    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cxlj6" event={"ID":"54d8f078-4001-4193-8c85-2951ebc88278","Type":"ContainerStarted","Data":"6fcc5aeb23d910ebbb2657ef05c39f903d6aa12b229e4bc083cc1f9b65719590"}
Sep 16 13:34:16.356364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.356112    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9gskl" event={"ID":"7dbd602d-d5b0-479e-8f26-ba2997c80f60","Type":"ContainerStarted","Data":"d357535f1cde557f11c3abb4843a338b05a0a6ba785878a05fe027e20565c9fd"}
Sep 16 13:34:16.356364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.356158    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9gskl" event={"ID":"7dbd602d-d5b0-479e-8f26-ba2997c80f60","Type":"ContainerStarted","Data":"03f099e6ad4baa90b1c941d550de7090e4d8c1fdf50bd03de129eaf480a1d51c"}
Sep 16 13:34:16.356364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.356277    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:34:16.358332 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.358302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qcnrn" event={"ID":"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37","Type":"ContainerStarted","Data":"b50fc41e3490d810ef7cab9ce09d8cb224bc14a6adc74d0f90fd2e917b96cbb7"}
Sep 16 13:34:16.371376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.371326    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9gskl" podStartSLOduration=66.371306876 podStartE2EDuration="1m6.371306876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:16.370017127 +0000 UTC m=+66.835954033" watchObservedRunningTime="2026-09-16 13:34:16.371306876 +0000 UTC m=+66.837243785"
Sep 16 13:34:16.528499 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.528471    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nnkqw\""
Sep 16 13:34:16.537207 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.537147    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xcljx"
Sep 16 13:34:16.876509 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:16.876277    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xcljx"]
Sep 16 13:34:16.907753 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:16.907721    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27eed164_faa1_4a17_ab02_a5f6528389bb.slice/crio-7e1c6644c7908d10643cfcb9a1a459b1aea635b81fc2419fbde1df1545a6d838 WatchSource:0}: Error finding container 7e1c6644c7908d10643cfcb9a1a459b1aea635b81fc2419fbde1df1545a6d838: Status 404 returned error can't find the container with id 7e1c6644c7908d10643cfcb9a1a459b1aea635b81fc2419fbde1df1545a6d838
Sep 16 13:34:17.364014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:17.363940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xcljx" event={"ID":"27eed164-faa1-4a17-ab02-a5f6528389bb","Type":"ContainerStarted","Data":"7e1c6644c7908d10643cfcb9a1a459b1aea635b81fc2419fbde1df1545a6d838"}
Sep 16 13:34:17.366451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:17.366412    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w2s9" event={"ID":"2786ffeb-964f-42fe-995f-e3be60ecd47f","Type":"ContainerStarted","Data":"e8c7093a085e52e9321f6009e84bfcd94e8cdfb012b1d9f03e52e0d50f2e31a6"}
Sep 16 13:34:17.366570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:17.366466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w2s9" event={"ID":"2786ffeb-964f-42fe-995f-e3be60ecd47f","Type":"ContainerStarted","Data":"b81fc2cc218d0c952cbb4dbbbfcf62dca6d0ada882379e9d79fcddd0f99afeef"}
Sep 16 13:34:17.383720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:17.383626    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8w2s9" podStartSLOduration=65.819714831 podStartE2EDuration="1m7.38360978s" totalImagesPullingTime="1.563894949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:17.382159629 +0000 UTC m=+67.848096581" watchObservedRunningTime="2026-09-16 13:34:17.38360978 +0000 UTC m=+67.849546715"
Sep 16 13:34:18.373107 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:18.373049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qcnrn" event={"ID":"e9af322c-e51f-4ca3-ae75-ac4dc4e7da37","Type":"ContainerStarted","Data":"4d32852097275a8202dbe76c2caf979d1c579906923c88d4d1f28056c9e58f15"}
Sep 16 13:34:18.392172 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:18.392106    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qcnrn" podStartSLOduration=17.628848536 podStartE2EDuration="20.392071283s" totalImagesPullingTime="2.763222747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:18.390771584 +0000 UTC m=+68.856708520" watchObservedRunningTime="2026-09-16 13:34:18.392071283 +0000 UTC m=+68.858008193"
Sep 16 13:34:19.316885 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.316849    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:19.428012 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.427979    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:19.615618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.615526    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:19.859731 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.859699    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:19.950134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.948959    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"]
Sep 16 13:34:19.951324 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.951292    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:19.955970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.955867    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 13:34:19.955970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.955867    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lbchk\""
Sep 16 13:34:19.955970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.955900    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 13:34:19.963234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:19.963210    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"]
Sep 16 13:34:20.009623 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.009591    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62b3e5d8-9b13-49cc-98ad-b058ac68c5f7-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-lngmg\" (UID: \"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.009769 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.009630    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62b3e5d8-9b13-49cc-98ad-b058ac68c5f7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lngmg\" (UID: \"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.012891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.012862    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c65897dc6-5b5f7"]
Sep 16 13:34:20.014994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.014966    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.017463 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.017439    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 13:34:20.017617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.017492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 13:34:20.017816 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.017796    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c5bs9\""
Sep 16 13:34:20.018038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.018020    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 13:34:20.024154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.024136    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 13:34:20.030975 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.030856    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c65897dc6-5b5f7"]
Sep 16 13:34:20.047371 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.047256    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:20.110521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-installation-pull-secrets\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110529    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62b3e5d8-9b13-49cc-98ad-b058ac68c5f7-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-lngmg\" (UID: \"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.110653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62b3e5d8-9b13-49cc-98ad-b058ac68c5f7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lngmg\" (UID: \"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.110653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110570    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-trusted-ca\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-registry-certificates\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fd5p\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-kube-api-access-5fd5p\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-registry-tls\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-bound-sa-token\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110750    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-ca-trust-extracted\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.110858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.110780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-image-registry-private-configuration\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.111403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.111380    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/62b3e5d8-9b13-49cc-98ad-b058ac68c5f7-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-lngmg\" (UID: \"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.113583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.113565    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/62b3e5d8-9b13-49cc-98ad-b058ac68c5f7-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-lngmg\" (UID: \"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.136857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.136829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:20.211626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211588    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fd5p\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-kube-api-access-5fd5p\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-registry-tls\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211654    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-bound-sa-token\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211697    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-ca-trust-extracted\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211731    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-image-registry-private-configuration\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211935 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-installation-pull-secrets\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211935 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-trusted-ca\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.211935 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.211863    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-registry-certificates\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.212888 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.212731    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-registry-certificates\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.213538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.213339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-ca-trust-extracted\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.214332 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.214292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-trusted-ca\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.215763 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.215265    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-installation-pull-secrets\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.215763 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.215722    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-registry-tls\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.216266 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.216248    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-image-registry-private-configuration\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.226856 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.226818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-bound-sa-token\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.227572 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.227550    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fd5p\" (UniqueName: \"kubernetes.io/projected/1a6f5ee2-d016-4e1e-9497-6a2bce11f846-kube-api-access-5fd5p\") pod \"image-registry-c65897dc6-5b5f7\" (UID: \"1a6f5ee2-d016-4e1e-9497-6a2bce11f846\") " pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.252664 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.252636    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:20.262898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.262880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"
Sep 16 13:34:20.329535 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.329275    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:20.337847 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.337819    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:20.389246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.389208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xcljx" event={"ID":"27eed164-faa1-4a17-ab02-a5f6528389bb","Type":"ContainerStarted","Data":"59ba8f65e22538cb8aba66838a8fd7e82ebb490dd8b82531f81a43b241aefd9e"}
Sep 16 13:34:20.389383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.389273    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xcljx" event={"ID":"27eed164-faa1-4a17-ab02-a5f6528389bb","Type":"ContainerStarted","Data":"23db679e64d7836af9afa8d86276867771b4f22c6b2ddf1f06a9785b3758bf87"}
Sep 16 13:34:20.389383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.389319    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xcljx"
Sep 16 13:34:20.390659 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.390637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cxlj6" event={"ID":"54d8f078-4001-4193-8c85-2951ebc88278","Type":"ContainerStarted","Data":"e077fabebe8c070d36cdd93496bdc1ff70a7c142172814af9b3bc2f675365122"}
Sep 16 13:34:20.410258 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.410211    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xcljx" podStartSLOduration=33.325195431 podStartE2EDuration="36.410192044s" totalImagesPullingTime="3.084996613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:20.410136479 +0000 UTC m=+70.876073410" watchObservedRunningTime="2026-09-16 13:34:20.410192044 +0000 UTC m=+70.876128954"
Sep 16 13:34:20.423022 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.422986    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cxlj6" podStartSLOduration=65.76205382 podStartE2EDuration="1m10.422972497s" totalImagesPullingTime="4.660918677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:20.422707975 +0000 UTC m=+70.888644900" watchObservedRunningTime="2026-09-16 13:34:20.422972497 +0000 UTC m=+70.888909405"
Sep 16 13:34:20.538626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.538597    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg"]
Sep 16 13:34:20.540894 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:20.540868    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b3e5d8_9b13_49cc_98ad_b058ac68c5f7.slice/crio-73b07161d34f18de0cbaf7c3cab6d6fce3b5433ce559c827141962d792db14fb WatchSource:0}: Error finding container 73b07161d34f18de0cbaf7c3cab6d6fce3b5433ce559c827141962d792db14fb: Status 404 returned error can't find the container with id 73b07161d34f18de0cbaf7c3cab6d6fce3b5433ce559c827141962d792db14fb
Sep 16 13:34:20.583450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.583421    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:20.604313 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.604283    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c65897dc6-5b5f7"]
Sep 16 13:34:20.607602 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:20.607580    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6f5ee2_d016_4e1e_9497_6a2bce11f846.slice/crio-305770e95cbfa28ba126d4dfc0abf69451855cd51793a5cccef2f74be54d5695 WatchSource:0}: Error finding container 305770e95cbfa28ba126d4dfc0abf69451855cd51793a5cccef2f74be54d5695: Status 404 returned error can't find the container with id 305770e95cbfa28ba126d4dfc0abf69451855cd51793a5cccef2f74be54d5695
Sep 16 13:34:20.683570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:20.683536    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:21.128822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.128777    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:21.229317 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.229276    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:21.329413 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.329382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:21.395847 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.395750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7" event={"ID":"1a6f5ee2-d016-4e1e-9497-6a2bce11f846","Type":"ContainerStarted","Data":"2cd6a5a0a708d8f84915eb63077755696c97f6e155d15cb64ca2ea6f9c6b0d05"}
Sep 16 13:34:21.395847 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.395812    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7" event={"ID":"1a6f5ee2-d016-4e1e-9497-6a2bce11f846","Type":"ContainerStarted","Data":"305770e95cbfa28ba126d4dfc0abf69451855cd51793a5cccef2f74be54d5695"}
Sep 16 13:34:21.396106 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.395857    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:21.397012 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.396986    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg" event={"ID":"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7","Type":"ContainerStarted","Data":"73b07161d34f18de0cbaf7c3cab6d6fce3b5433ce559c827141962d792db14fb"}
Sep 16 13:34:21.414508 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.414463    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7" podStartSLOduration=2.414447256 podStartE2EDuration="2.414447256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:21.413650277 +0000 UTC m=+71.879587200" watchObservedRunningTime="2026-09-16 13:34:21.414447256 +0000 UTC m=+71.880384164"
Sep 16 13:34:21.553245 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.553214    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:21.615617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:21.615580    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:22.220197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:22.220159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:22.279583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:22.279546    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:23.405653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:23.405623    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg" event={"ID":"62b3e5d8-9b13-49cc-98ad-b058ac68c5f7","Type":"ContainerStarted","Data":"e6eba681e7d74fd5967560fe668f6b44ab3011fee0f40f7d722bca0c7db1a03d"}
Sep 16 13:34:23.421122 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:23.421051    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-lngmg" podStartSLOduration=2.337558337 podStartE2EDuration="4.421031888s" totalImagesPullingTime="2.083473551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:23.420136076 +0000 UTC m=+73.886072995" watchObservedRunningTime="2026-09-16 13:34:23.421031888 +0000 UTC m=+73.886968796"
Sep 16 13:34:23.424550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:23.424526    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:23.524935 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:23.524901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:23.621184 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:23.621151    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:25.352518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.352461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:34:25.355075 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.355050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/68537ef5-22fb-4534-9627-5441ca8ac4a5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-bzw76\" (UID: \"68537ef5-22fb-4534-9627-5441ca8ac4a5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:34:25.452956 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.452919    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:25.453548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.453528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d999b14-0ce5-49e5-8c6e-3c74fedd210c-service-ca-bundle\") pod \"router-default-5b79cff5dd-mnnh2\" (UID: \"8d999b14-0ce5-49e5-8c6e-3c74fedd210c\") " pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:25.514810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.514778    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vnpgd\""
Sep 16 13:34:25.523401 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.523367    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"
Sep 16 13:34:25.639880 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.639762    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xldgt\""
Sep 16 13:34:25.648433 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.648415    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:25.791789 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.791758    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76"]
Sep 16 13:34:25.794746 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:25.794719    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68537ef5_22fb_4534_9627_5441ca8ac4a5.slice/crio-761c5ebf29074d45fc1188c70ac1ae1cf48ab85fe695ee705d231252f60dff10 WatchSource:0}: Error finding container 761c5ebf29074d45fc1188c70ac1ae1cf48ab85fe695ee705d231252f60dff10: Status 404 returned error can't find the container with id 761c5ebf29074d45fc1188c70ac1ae1cf48ab85fe695ee705d231252f60dff10
Sep 16 13:34:25.916795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:25.916775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b79cff5dd-mnnh2"]
Sep 16 13:34:25.919377 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:25.919351    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d999b14_0ce5_49e5_8c6e_3c74fedd210c.slice/crio-878075a87c1195335dcb4f660dac2dc696ec6c7b4286064db7f3e0e7393ce98d WatchSource:0}: Error finding container 878075a87c1195335dcb4f660dac2dc696ec6c7b4286064db7f3e0e7393ce98d: Status 404 returned error can't find the container with id 878075a87c1195335dcb4f660dac2dc696ec6c7b4286064db7f3e0e7393ce98d
Sep 16 13:34:26.418490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:26.418457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2" event={"ID":"8d999b14-0ce5-49e5-8c6e-3c74fedd210c","Type":"ContainerStarted","Data":"27db797f94bbb79c7474b07c8c6cfae9bf3b7bb2a01209a0857709d20b1dcf53"}
Sep 16 13:34:26.418899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:26.418509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2" event={"ID":"8d999b14-0ce5-49e5-8c6e-3c74fedd210c","Type":"ContainerStarted","Data":"878075a87c1195335dcb4f660dac2dc696ec6c7b4286064db7f3e0e7393ce98d"}
Sep 16 13:34:26.419713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:26.419692    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76" event={"ID":"68537ef5-22fb-4534-9627-5441ca8ac4a5","Type":"ContainerStarted","Data":"761c5ebf29074d45fc1188c70ac1ae1cf48ab85fe695ee705d231252f60dff10"}
Sep 16 13:34:26.438700 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:26.438655    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2" podStartSLOduration=33.438639993 podStartE2EDuration="33.438639993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:26.43735275 +0000 UTC m=+76.903289692" watchObservedRunningTime="2026-09-16 13:34:26.438639993 +0000 UTC m=+76.904576897"
Sep 16 13:34:26.649463 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:26.649423    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:26.652185 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:26.652147    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:27.021726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:27.021688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:27.076680 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:27.076648    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:27.424129 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:27.424083    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:27.425483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:27.425460    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b79cff5dd-mnnh2"
Sep 16 13:34:28.044503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:28.044426    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:28.141126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:28.141082    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:28.212006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:28.211953    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:28.354730 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:28.354640    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:28.428295 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:28.428263    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76" event={"ID":"68537ef5-22fb-4534-9627-5441ca8ac4a5","Type":"ContainerStarted","Data":"385d747bad83a8bf71a98fb156849ba5b89138bb23bdea71f91080d70dee689a"}
Sep 16 13:34:28.446210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:28.446165    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-bzw76" podStartSLOduration=33.510173021 podStartE2EDuration="35.446150706s" totalImagesPullingTime="1.935977685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:28.444882528 +0000 UTC m=+78.910819460" watchObservedRunningTime="2026-09-16 13:34:28.446150706 +0000 UTC m=+78.912087640"
Sep 16 13:34:29.951570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:29.951517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:30.003545 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:30.003512    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:30.399269 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:30.399178    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xcljx"
Sep 16 13:34:32.231171 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:32.231043    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:32.331639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:32.331607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:32.432675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:32.432646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:33.442920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:33.442884    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:33.927485 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:33.927456    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:34.832517 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:34.832486    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:35.780821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:35.780788    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:35.846354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:35.846305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.050660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.050578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.102874 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.102813    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.377787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.377703    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.482290 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.482258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.588002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.587968    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.627694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.627662    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"]
Sep 16 13:34:36.630967 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.630902    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2nqnw"]
Sep 16 13:34:36.631133 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.631046    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.633346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.633322    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 13:34:36.633481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.633324    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 13:34:36.633674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.633656    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.633744 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.633685    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 13:34:36.633744 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.633718    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 13:34:36.633744 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.633723    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-48h56\""
Sep 16 13:34:36.635609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.635592    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 13:34:36.635609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.635602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 13:34:36.635779 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.635751    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bdhnc\""
Sep 16 13:34:36.635881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.635864    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 13:34:36.642401 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.642381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"]
Sep 16 13:34:36.667621 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.667589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.735554 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.735520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-wtmp\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.735912 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.735880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbce715-10b6-4573-acde-12b63cd9beb4-metrics-client-ca\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736033 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.735939    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.736033 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.735959    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-sys\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736033 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.735992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.736033 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-textfile\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3258470a-1466-40e0-afca-08530a9a88e9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.736223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-root\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-tls\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3258470a-1466-40e0-afca-08530a9a88e9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.736387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssvt\" (UniqueName: \"kubernetes.io/projected/ebbce715-10b6-4573-acde-12b63cd9beb4-kube-api-access-zssvt\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.736387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.736387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.736316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bz6\" (UniqueName: \"kubernetes.io/projected/3258470a-1466-40e0-afca-08530a9a88e9-kube-api-access-f2bz6\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.783425 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.783392    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.836761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836730    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-tls\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.836761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3258470a-1466-40e0-afca-08530a9a88e9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836816    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssvt\" (UniqueName: \"kubernetes.io/projected/ebbce715-10b6-4573-acde-12b63cd9beb4-kube-api-access-zssvt\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bz6\" (UniqueName: \"kubernetes.io/projected/3258470a-1466-40e0-afca-08530a9a88e9-kube-api-access-f2bz6\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-wtmp\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.836989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbce715-10b6-4573-acde-12b63cd9beb4-metrics-client-ca\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-sys\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837116    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-textfile\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837146    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3258470a-1466-40e0-afca-08530a9a88e9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-root\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.837264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-root\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:36.837372    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 13:34:36.837431 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:34:36.837437    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-tls podName:ebbce715-10b6-4573-acde-12b63cd9beb4 nodeName:}" failed. No retries permitted until 2026-09-16 13:34:37.337416525 +0000 UTC m=+87.803353427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-tls") pod "node-exporter-2nqnw" (UID: "ebbce715-10b6-4573-acde-12b63cd9beb4") : secret "node-exporter-tls" not found
Sep 16 13:34:36.838904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.838379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-accelerators-collector-config\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.838904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.838621    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-textfile\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.838904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.838718    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.838904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.838857    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-wtmp\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.839431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.839224    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebbce715-10b6-4573-acde-12b63cd9beb4-sys\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.839431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.839358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebbce715-10b6-4573-acde-12b63cd9beb4-metrics-client-ca\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.839742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.839697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3258470a-1466-40e0-afca-08530a9a88e9-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.839867 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.839844    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3258470a-1466-40e0-afca-08530a9a88e9-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.843841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.843498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.843841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.843613    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3258470a-1466-40e0-afca-08530a9a88e9-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.843841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.843772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.847963 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.847944    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bz6\" (UniqueName: \"kubernetes.io/projected/3258470a-1466-40e0-afca-08530a9a88e9-kube-api-access-f2bz6\") pod \"kube-state-metrics-59fbdd77d-lm26r\" (UID: \"3258470a-1466-40e0-afca-08530a9a88e9\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:36.848694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.848669    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssvt\" (UniqueName: \"kubernetes.io/projected/ebbce715-10b6-4573-acde-12b63cd9beb4-kube-api-access-zssvt\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:36.884145 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.884049    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:36.943721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:36.943686    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"
Sep 16 13:34:37.116480 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.116448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:37.218665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.218634    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r"]
Sep 16 13:34:37.220806 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.220775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:37.222408 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:37.222372    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3258470a_1466_40e0_afca_08530a9a88e9.slice/crio-9cde0c05a2a25c6a6c63745a0139367a9e0f842a6da3f86a43d58a053c5c8f8f WatchSource:0}: Error finding container 9cde0c05a2a25c6a6c63745a0139367a9e0f842a6da3f86a43d58a053c5c8f8f: Status 404 returned error can't find the container with id 9cde0c05a2a25c6a6c63745a0139367a9e0f842a6da3f86a43d58a053c5c8f8f
Sep 16 13:34:37.340856 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.340823    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-tls\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:37.343366 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.343342    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ebbce715-10b6-4573-acde-12b63cd9beb4-node-exporter-tls\") pod \"node-exporter-2nqnw\" (UID: \"ebbce715-10b6-4573-acde-12b63cd9beb4\") " pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:37.395522 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.395484    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:37.457262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.457221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r" event={"ID":"3258470a-1466-40e0-afca-08530a9a88e9","Type":"ContainerStarted","Data":"9cde0c05a2a25c6a6c63745a0139367a9e0f842a6da3f86a43d58a053c5c8f8f"}
Sep 16 13:34:37.549703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.549615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2nqnw"
Sep 16 13:34:37.566315 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:37.566282    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebbce715_10b6_4573_acde_12b63cd9beb4.slice/crio-3303377a21037096fec3b16bb9d6d0d5b479d6889fcc0cecf991174a2c5bbc4f WatchSource:0}: Error finding container 3303377a21037096fec3b16bb9d6d0d5b479d6889fcc0cecf991174a2c5bbc4f: Status 404 returned error can't find the container with id 3303377a21037096fec3b16bb9d6d0d5b479d6889fcc0cecf991174a2c5bbc4f
Sep 16 13:34:37.589890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.589861    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:37.670698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.670652    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:37.734715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:37.734679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:38.437392 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:38.437356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:38.463042 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:38.463003    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nqnw" event={"ID":"ebbce715-10b6-4573-acde-12b63cd9beb4","Type":"ContainerStarted","Data":"3303377a21037096fec3b16bb9d6d0d5b479d6889fcc0cecf991174a2c5bbc4f"}
Sep 16 13:34:38.539307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:38.539263    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:39.469395 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.469364    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r" event={"ID":"3258470a-1466-40e0-afca-08530a9a88e9","Type":"ContainerStarted","Data":"17279e301ad6f35bacac778010a9a3f52e721fcffa69f6d93c97268cce88e4bf"}
Sep 16 13:34:39.469869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.469424    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r" event={"ID":"3258470a-1466-40e0-afca-08530a9a88e9","Type":"ContainerStarted","Data":"3270bca95c9afe67cd20ccf43616f721cccf675e69d287e786358e1890c1b94a"}
Sep 16 13:34:39.469869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.469453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r" event={"ID":"3258470a-1466-40e0-afca-08530a9a88e9","Type":"ContainerStarted","Data":"799856d115980d7a6497518f3ca3dee6266148675f4b50a5a215929eb532aa82"}
Sep 16 13:34:39.470950 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.470928    2556 generic.go:356] "Generic (PLEG): container finished" podID="ebbce715-10b6-4573-acde-12b63cd9beb4" containerID="e63968e3454f613c953dcac75d5c59d5545c16a7c1ec3b2e38649cefed5f5d63" exitCode=0
Sep 16 13:34:39.471048 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.470990    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nqnw" event={"ID":"ebbce715-10b6-4573-acde-12b63cd9beb4","Type":"ContainerDied","Data":"e63968e3454f613c953dcac75d5c59d5545c16a7c1ec3b2e38649cefed5f5d63"}
Sep 16 13:34:39.487863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.487817    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-lm26r" podStartSLOduration=1.885316029 podStartE2EDuration="3.487799804s" totalImagesPullingTime="1.602483775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:39.485860777 +0000 UTC m=+89.951797685" watchObservedRunningTime="2026-09-16 13:34:39.487799804 +0000 UTC m=+89.953736714"
Sep 16 13:34:39.531808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.531772    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:39.642834 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.642794    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:39.672908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.672872    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f89c58984-jfr88"]
Sep 16 13:34:39.676479 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.676456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.679038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679012    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 13:34:39.679329 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 13:34:39.679431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679353    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4kfbveako0mkt\""
Sep 16 13:34:39.679431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 13:34:39.679431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679402    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 13:34:39.679586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679430    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ffkrp\""
Sep 16 13:34:39.679586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.679507    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 13:34:39.694565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.694532    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f89c58984-jfr88"]
Sep 16 13:34:39.721040 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.720953    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763214    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmczg\" (UniqueName: \"kubernetes.io/projected/422ce74c-8838-43fa-91db-1a29dae00fcb-kube-api-access-bmczg\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763268    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-grpc-tls\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422ce74c-8838-43fa-91db-1a29dae00fcb-metrics-client-ca\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763358    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-tls\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.763588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.763438    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.863868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.863832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.863868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.863869    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.864180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.863886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmczg\" (UniqueName: \"kubernetes.io/projected/422ce74c-8838-43fa-91db-1a29dae00fcb-kube-api-access-bmczg\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.864180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.863903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-grpc-tls\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.864180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.863953    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422ce74c-8838-43fa-91db-1a29dae00fcb-metrics-client-ca\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.864180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.863978    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.864180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.864015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-tls\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.864180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.864044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.865946 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.865357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/422ce74c-8838-43fa-91db-1a29dae00fcb-metrics-client-ca\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.867318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.867221    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.867318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.867263    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-grpc-tls\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.867499 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.867324    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.867660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.867552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.867660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.867649    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.867971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.867949    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/422ce74c-8838-43fa-91db-1a29dae00fcb-secret-thanos-querier-tls\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.872444 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.872425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmczg\" (UniqueName: \"kubernetes.io/projected/422ce74c-8838-43fa-91db-1a29dae00fcb-kube-api-access-bmczg\") pod \"thanos-querier-5f89c58984-jfr88\" (UID: \"422ce74c-8838-43fa-91db-1a29dae00fcb\") " pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:39.908638 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.908607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:39.988138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:39.988031    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:40.275047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:40.274930    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f89c58984-jfr88"]
Sep 16 13:34:40.277592 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:40.277562    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422ce74c_8838_43fa_91db_1a29dae00fcb.slice/crio-856a1c601712ea41ed67a126194c2e2665c0885abfd2134e289cabbf887faeb9 WatchSource:0}: Error finding container 856a1c601712ea41ed67a126194c2e2665c0885abfd2134e289cabbf887faeb9: Status 404 returned error can't find the container with id 856a1c601712ea41ed67a126194c2e2665c0885abfd2134e289cabbf887faeb9
Sep 16 13:34:40.304468 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:40.304438    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:40.476311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:40.476274    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"856a1c601712ea41ed67a126194c2e2665c0885abfd2134e289cabbf887faeb9"}
Sep 16 13:34:40.478493 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:40.478463    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nqnw" event={"ID":"ebbce715-10b6-4573-acde-12b63cd9beb4","Type":"ContainerStarted","Data":"4aca79440f62ca132d62ea0b3d2a47334244f842d45b930c318a2a376a771d40"}
Sep 16 13:34:40.478623 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:40.478522    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2nqnw" event={"ID":"ebbce715-10b6-4573-acde-12b63cd9beb4","Type":"ContainerStarted","Data":"32b331401ac2b46ff510cf567f6cdf58549bc98580dc7877c164d23e069112e0"}
Sep 16 13:34:40.497315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:40.497263    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2nqnw" podStartSLOduration=3.139219869 podStartE2EDuration="4.497245434s" totalImagesPullingTime="1.261357003s" totalInitContainerRuntime="96.668562ms" isStatefulPod=true podCreationTimestamp="2026-09-16 13:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:40.495452405 +0000 UTC m=+90.961389314" watchObservedRunningTime="2026-09-16 13:34:40.497245434 +0000 UTC m=+90.963182355"
Sep 16 13:34:41.417818 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:41.417770    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:41.515179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:41.515142    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:41.634733 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:41.634679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:42.405611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:42.405582    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c65897dc6-5b5f7"
Sep 16 13:34:42.582862 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:42.582831    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:42.684369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:42.684333    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:42.783381 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:42.783350    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:42.883212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:42.883178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:42.987082 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:42.987044    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:43.494416 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:43.494380    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"d86bf6310f131a3b5148fef38481280268ca1e6c7c525ff25a35c360c6a5c4bf"}
Sep 16 13:34:43.494590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:43.494487    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"16537566d6822eb26161fed599bb66c1c6d8f74c2b653d27f7204d374869a156"}
Sep 16 13:34:43.494590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:43.494519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"9b7cdf653a532d0c9731392b6b5f72dfe421c2ceff64e7b41b7c29dff1e95ddc"}
Sep 16 13:34:44.351780 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.351745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:44.407395 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.407357    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:44.500848 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.500811    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"c1c1cbddfe41191e30df49f6379591b3291079afb5b4527d2a8c5f1ffb3177be"}
Sep 16 13:34:44.501037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.500891    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"fc1b5d3aad62c3c7c8f51c5ad76736e1d96156e1031ba1f568d78af09414b562"}
Sep 16 13:34:44.501037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.500934    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" event={"ID":"422ce74c-8838-43fa-91db-1a29dae00fcb","Type":"ContainerStarted","Data":"9b8827ef9e376f14deb611dc8646ca7d5421e9bb3bc50a1513c89586e3f85cb4"}
Sep 16 13:34:44.501037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.500975    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:44.527113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.527042    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88" podStartSLOduration=2.390936929 podStartE2EDuration="5.527026079s" totalImagesPullingTime="3.13608915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:44.525644818 +0000 UTC m=+94.991581726" watchObservedRunningTime="2026-09-16 13:34:44.527026079 +0000 UTC m=+94.992962987"
Sep 16 13:34:44.882018 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.881980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:44.981941 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:44.981908    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:46.234550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.234515    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75dbcfdf6c-mwsn6"]
Sep 16 13:34:46.239816 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.239798    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.241890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.241869    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 13:34:46.241890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.241883    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 13:34:46.242061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.241898    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 13:34:46.242491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.242473    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 13:34:46.242491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.242488    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 13:34:46.242713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.242492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q7ntl\""
Sep 16 13:34:46.242713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.242473    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 13:34:46.242713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.242492    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 13:34:46.247470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.247447    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75dbcfdf6c-mwsn6"]
Sep 16 13:34:46.248465 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.248444    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 13:34:46.318621 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318573    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-oauth-serving-cert\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.318621 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-trusted-ca-bundle\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.318889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-config\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.318889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318720    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-service-ca\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.318889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-oauth-config\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.318889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c584\" (UniqueName: \"kubernetes.io/projected/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-kube-api-access-6c584\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.318889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.318809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-serving-cert\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-oauth-config\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420211    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c584\" (UniqueName: \"kubernetes.io/projected/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-kube-api-access-6c584\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420229    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-serving-cert\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420260    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-oauth-serving-cert\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420277    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-trusted-ca-bundle\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-config\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.420477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.420342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-service-ca\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.421229 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.421201    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-oauth-serving-cert\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.421353 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.421234    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-config\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.421422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.421234    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-service-ca\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.421422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.421279    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-trusted-ca-bundle\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.422882 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.422860    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-serving-cert\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.422929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.422895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-oauth-config\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.427424 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.427404    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c584\" (UniqueName: \"kubernetes.io/projected/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-kube-api-access-6c584\") pod \"console-75dbcfdf6c-mwsn6\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") " pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.551142 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.551037    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:46.823368 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:46.823343    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75dbcfdf6c-mwsn6"]
Sep 16 13:34:46.825842 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:34:46.825812    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca0c8e0_28da_4a56_b4ba_d53160b8c9ce.slice/crio-8026e43a7373459e3054d84ba0196bcef05faf16ca03b88e19831bf923584faf WatchSource:0}: Error finding container 8026e43a7373459e3054d84ba0196bcef05faf16ca03b88e19831bf923584faf: Status 404 returned error can't find the container with id 8026e43a7373459e3054d84ba0196bcef05faf16ca03b88e19831bf923584faf
Sep 16 13:34:47.369397 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:47.369361    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9gskl"
Sep 16 13:34:47.513019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:47.512988    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dbcfdf6c-mwsn6" event={"ID":"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce","Type":"ContainerStarted","Data":"8026e43a7373459e3054d84ba0196bcef05faf16ca03b88e19831bf923584faf"}
Sep 16 13:34:50.510147 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:50.510118    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f89c58984-jfr88"
Sep 16 13:34:50.528030 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:50.528000    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dbcfdf6c-mwsn6" event={"ID":"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce","Type":"ContainerStarted","Data":"c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4"}
Sep 16 13:34:50.555621 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:50.555573    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75dbcfdf6c-mwsn6" podStartSLOduration=1.7702285500000001 podStartE2EDuration="4.555555304s" totalImagesPullingTime="2.785326754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:34:50.554472042 +0000 UTC m=+101.020408963" watchObservedRunningTime="2026-09-16 13:34:50.555555304 +0000 UTC m=+101.021492211"
Sep 16 13:34:51.532679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:51.532646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:56.551312 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:56.551284    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:56.551799 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:56.551383    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:56.556213 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:56.556189    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:57.117670 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.117637    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:57.171245 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.171195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:57.527133 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.527066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:57.559648 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.559622    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:34:57.580677 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.580647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:57.711778 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.711696    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:34:57.812379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:34:57.812265    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:04.729721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:04.729674    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79cff5dd-mnnh2_8d999b14-0ce5-49e5-8c6e-3c74fedd210c/router/0.log"
Sep 16 13:35:04.741971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:04.741946    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bj9zd_fa9b128f-810f-4a59-9ce9-c74bbe571128/serve-healthcheck-canary/0.log"
Sep 16 13:35:05.454647 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:05.454581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:05.665498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:05.665466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:08.419078 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:08.419045    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:08.519624 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:08.519595    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:08.596767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:08.596732    2556 generic.go:356] "Generic (PLEG): container finished" podID="382a1e6d-8b4a-4948-82b0-223d7a8d9564" containerID="5ed9266f7f4c206951fd1bec0b3a585c10f0f45f51965a67e52124975c05fe8d" exitCode=0
Sep 16 13:35:08.596950 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:08.596807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625" event={"ID":"382a1e6d-8b4a-4948-82b0-223d7a8d9564","Type":"ContainerDied","Data":"5ed9266f7f4c206951fd1bec0b3a585c10f0f45f51965a67e52124975c05fe8d"}
Sep 16 13:35:08.597168 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:08.597154    2556 scope.go:117] "RemoveContainer" containerID="5ed9266f7f4c206951fd1bec0b3a585c10f0f45f51965a67e52124975c05fe8d"
Sep 16 13:35:09.602875 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:09.602841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-kg625" event={"ID":"382a1e6d-8b4a-4948-82b0-223d7a8d9564","Type":"ContainerStarted","Data":"4efc8a18031b7e48ea7f103cf61acfba4b5f88ae3dede583b715d97d5bdbe015"}
Sep 16 13:35:14.742198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:14.742164    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:14.842619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:14.842579    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:14.897685 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:14.897652    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:23.415998 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:23.415968    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:23.468017 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:23.467987    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:29.681585 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:29.681546    2556 generic.go:356] "Generic (PLEG): container finished" podID="d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3" containerID="8688d17f9146c5ed2fa37b5a44e19109840ae504e55b63a205e9cf7cfe1f6653" exitCode=0
Sep 16 13:35:29.681960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:29.681613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr" event={"ID":"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3","Type":"ContainerDied","Data":"8688d17f9146c5ed2fa37b5a44e19109840ae504e55b63a205e9cf7cfe1f6653"}
Sep 16 13:35:29.682042 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:29.682023    2556 scope.go:117] "RemoveContainer" containerID="8688d17f9146c5ed2fa37b5a44e19109840ae504e55b63a205e9cf7cfe1f6653"
Sep 16 13:35:30.687679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:30.687645    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b4lwr" event={"ID":"d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3","Type":"ContainerStarted","Data":"a74cf39324809ef5e8393087984d25f1333287dd8b9b19c83cf4e96ab6b4be9c"}
Sep 16 13:35:55.849047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:55.849012    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:35:56.352179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:35:56.352142    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:00.988450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.988412    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79cd7db795-9hmgk"]
Sep 16 13:36:00.995821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.995793    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:00.997978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.997951    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dc7cp\""
Sep 16 13:36:00.997978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.997960    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 13:36:00.998188 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.998011    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 13:36:00.998188 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.997962    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 13:36:00.998188 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.998129    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 13:36:00.998351 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:00.998335    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 13:36:01.003617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.003591    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 13:36:01.006627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.006602    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79cd7db795-9hmgk"]
Sep 16 13:36:01.099422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-telemeter-client-tls\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5qt\" (UniqueName: \"kubernetes.io/projected/b2852957-d8f2-4947-8eac-b9227c0110bc-kube-api-access-zx5qt\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-serving-certs-ca-bundle\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099560    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-secret-telemeter-client\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-metrics-client-ca\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.099805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.099682    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-federate-client-tls\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200457 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-serving-certs-ca-bundle\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-secret-telemeter-client\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200508    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200780 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-metrics-client-ca\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200780 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200759    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-federate-client-tls\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200885 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.200885 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.200860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-telemeter-client-tls\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.201470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.201439    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5qt\" (UniqueName: \"kubernetes.io/projected/b2852957-d8f2-4947-8eac-b9227c0110bc-kube-api-access-zx5qt\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.201669 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.201642    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-serving-certs-ca-bundle\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.202025 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.201996    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.202589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.202568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2852957-d8f2-4947-8eac-b9227c0110bc-metrics-client-ca\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.204320 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.204296    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-telemeter-client-tls\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.204490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.204434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.204490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.204449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-federate-client-tls\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.204650 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.203933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b2852957-d8f2-4947-8eac-b9227c0110bc-secret-telemeter-client\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.211361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.211339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5qt\" (UniqueName: \"kubernetes.io/projected/b2852957-d8f2-4947-8eac-b9227c0110bc-kube-api-access-zx5qt\") pod \"telemeter-client-79cd7db795-9hmgk\" (UID: \"b2852957-d8f2-4947-8eac-b9227c0110bc\") " pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.306347 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.306234    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk"
Sep 16 13:36:01.582121 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.582074    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79cd7db795-9hmgk"]
Sep 16 13:36:01.583843 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:36:01.583814    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2852957_d8f2_4947_8eac_b9227c0110bc.slice/crio-333521f0689b7e6f471bb0d9669690aa0332c6aa1ea6e6240180259d871a32e7 WatchSource:0}: Error finding container 333521f0689b7e6f471bb0d9669690aa0332c6aa1ea6e6240180259d871a32e7: Status 404 returned error can't find the container with id 333521f0689b7e6f471bb0d9669690aa0332c6aa1ea6e6240180259d871a32e7
Sep 16 13:36:01.807115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:01.807058    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk" event={"ID":"b2852957-d8f2-4947-8eac-b9227c0110bc","Type":"ContainerStarted","Data":"333521f0689b7e6f471bb0d9669690aa0332c6aa1ea6e6240180259d871a32e7"}
Sep 16 13:36:03.817436 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:03.817398    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk" event={"ID":"b2852957-d8f2-4947-8eac-b9227c0110bc","Type":"ContainerStarted","Data":"d671a1c77cd2316b96e5d48ae909507e3654915733948d3fb6ed931e8d9311e1"}
Sep 16 13:36:04.824391 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.824352    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk" event={"ID":"b2852957-d8f2-4947-8eac-b9227c0110bc","Type":"ContainerStarted","Data":"0fa8bc3bfd1e33603064b9113b1a1bc54d48036fbfece5946fa547d5a6266c1a"}
Sep 16 13:36:04.824776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.824413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk" event={"ID":"b2852957-d8f2-4947-8eac-b9227c0110bc","Type":"ContainerStarted","Data":"d223fecd532482112e791e4ec5d4320d1d8a6354886ecf1f533717c885d41a07"}
Sep 16 13:36:04.844492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.844432    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79cd7db795-9hmgk" podStartSLOduration=1.837515606 podStartE2EDuration="4.844414709s" totalImagesPullingTime="3.006899103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:04.842909653 +0000 UTC m=+175.308846591" watchObservedRunningTime="2026-09-16 13:36:04.844414709 +0000 UTC m=+175.310351616"
Sep 16 13:36:04.926621 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.926589    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8dc5df7fb-x6tkk"]
Sep 16 13:36:04.929698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.929680    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:04.940867 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.940836    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dc5df7fb-x6tkk"]
Sep 16 13:36:04.986218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:04.986181    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:05.035108 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035070    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-oauth-serving-cert\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.035282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035136    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-console-config\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.035282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5m65\" (UniqueName: \"kubernetes.io/projected/46419cdf-98d8-4605-9a3e-28f035548c6e-kube-api-access-h5m65\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.035282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-service-ca\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.035282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-serving-cert\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.035434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-trusted-ca-bundle\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.035434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.035341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-oauth-config\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.068638 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.068606    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:05.136441 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-oauth-config\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.136635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-oauth-serving-cert\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.136635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136476    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-console-config\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.136635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5m65\" (UniqueName: \"kubernetes.io/projected/46419cdf-98d8-4605-9a3e-28f035548c6e-kube-api-access-h5m65\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.136635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-service-ca\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.136635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136572    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-serving-cert\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.136635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.136601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-trusted-ca-bundle\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.137426 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.137399    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-oauth-serving-cert\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.137580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.137448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-console-config\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.137580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.137509    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-trusted-ca-bundle\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.137756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.137704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-service-ca\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.138166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.138144    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:05.139148 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.139129    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-oauth-config\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.139395 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.139373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-serving-cert\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.144177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.144151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5m65\" (UniqueName: \"kubernetes.io/projected/46419cdf-98d8-4605-9a3e-28f035548c6e-kube-api-access-h5m65\") pod \"console-8dc5df7fb-x6tkk\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") " pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.245428 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.245335    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:05.518373 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.518293    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dc5df7fb-x6tkk"]
Sep 16 13:36:05.521470 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:36:05.521439    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46419cdf_98d8_4605_9a3e_28f035548c6e.slice/crio-f1641b198ae6d80daecd88ed84ae4c67f28384cd4f974614164a78680ee81d17 WatchSource:0}: Error finding container f1641b198ae6d80daecd88ed84ae4c67f28384cd4f974614164a78680ee81d17: Status 404 returned error can't find the container with id f1641b198ae6d80daecd88ed84ae4c67f28384cd4f974614164a78680ee81d17
Sep 16 13:36:05.830957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.830848    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc5df7fb-x6tkk" event={"ID":"46419cdf-98d8-4605-9a3e-28f035548c6e","Type":"ContainerStarted","Data":"6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962"}
Sep 16 13:36:05.830957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.830904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc5df7fb-x6tkk" event={"ID":"46419cdf-98d8-4605-9a3e-28f035548c6e","Type":"ContainerStarted","Data":"f1641b198ae6d80daecd88ed84ae4c67f28384cd4f974614164a78680ee81d17"}
Sep 16 13:36:05.846813 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:05.846755    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8dc5df7fb-x6tkk" podStartSLOduration=1.846735317 podStartE2EDuration="1.846735317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:05.845894102 +0000 UTC m=+176.311831009" watchObservedRunningTime="2026-09-16 13:36:05.846735317 +0000 UTC m=+176.312672226"
Sep 16 13:36:06.422024 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:06.421994    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:06.472334 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:06.472298    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:09.136932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:09.136897    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:11.092109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:11.092060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:11.191744 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:11.191708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:13.120917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.120882    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dc5df7fb-x6tkk"]
Sep 16 13:36:13.148837 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.148808    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57785d5cf8-m929f"]
Sep 16 13:36:13.153102 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.153071    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.161208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.161185    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57785d5cf8-m929f"]
Sep 16 13:36:13.308496 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-trusted-ca-bundle\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.308686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308511    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-oauth-serving-cert\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.308686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-service-ca\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.308686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308594    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-config\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.308686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308635    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-serving-cert\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.308848 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-oauth-config\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.308848 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.308745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc64h\" (UniqueName: \"kubernetes.io/projected/7a1bd368-879b-49cb-a76a-7ff7c847d76a-kube-api-access-jc64h\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.319356 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.319322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:13.409537 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409448    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-oauth-config\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.409537 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409496    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc64h\" (UniqueName: \"kubernetes.io/projected/7a1bd368-879b-49cb-a76a-7ff7c847d76a-kube-api-access-jc64h\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.409718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-trusted-ca-bundle\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.409718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-oauth-serving-cert\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.409809 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-service-ca\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.409809 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-config\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.409809 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.409789    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-serving-cert\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.410616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.410523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-service-ca\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.410616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.410546    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-trusted-ca-bundle\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.410616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.410523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-oauth-serving-cert\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.410873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.410745    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-config\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.412310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.412285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-oauth-config\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.412407 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.412367    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-serving-cert\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.417068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.417045    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc64h\" (UniqueName: \"kubernetes.io/projected/7a1bd368-879b-49cb-a76a-7ff7c847d76a-kube-api-access-jc64h\") pod \"console-57785d5cf8-m929f\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") " pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.419727 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.419703    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:13.462538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.462506    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:13.737224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.737189    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57785d5cf8-m929f"]
Sep 16 13:36:13.740644 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:36:13.740609    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1bd368_879b_49cb_a76a_7ff7c847d76a.slice/crio-b4d6d95d323c3f403fa812d57e937603c89315d7e350f6054dc83dd8098d4e41 WatchSource:0}: Error finding container b4d6d95d323c3f403fa812d57e937603c89315d7e350f6054dc83dd8098d4e41: Status 404 returned error can't find the container with id b4d6d95d323c3f403fa812d57e937603c89315d7e350f6054dc83dd8098d4e41
Sep 16 13:36:13.865947 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.865907    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57785d5cf8-m929f" event={"ID":"7a1bd368-879b-49cb-a76a-7ff7c847d76a","Type":"ContainerStarted","Data":"ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d"}
Sep 16 13:36:13.866189 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.865957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57785d5cf8-m929f" event={"ID":"7a1bd368-879b-49cb-a76a-7ff7c847d76a","Type":"ContainerStarted","Data":"b4d6d95d323c3f403fa812d57e937603c89315d7e350f6054dc83dd8098d4e41"}
Sep 16 13:36:13.883404 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.883355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57785d5cf8-m929f" podStartSLOduration=0.883337797 podStartE2EDuration="883.337797ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:36:13.882614922 +0000 UTC m=+184.348551870" watchObservedRunningTime="2026-09-16 13:36:13.883337797 +0000 UTC m=+184.349274705"
Sep 16 13:36:13.915911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:13.915877    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:15.245468 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:15.245437    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:23.142396 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.142364    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:23.192748 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.192715    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:23.463234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.463184    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:23.463408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.463297    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:23.468277 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.468256    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:23.908040 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.907961    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:36:23.952306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:23.952270    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75dbcfdf6c-mwsn6"]
Sep 16 13:36:24.050625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:24.050578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:24.100890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:24.100855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:24.159283 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:24.159196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:34.910189 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:34.910154    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:38.140661 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.140623    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8dc5df7fb-x6tkk" podUID="46419cdf-98d8-4605-9a3e-28f035548c6e" containerName="console" containerID="cri-o://6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962" gracePeriod=15
Sep 16 13:36:38.376798 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.376774    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:38.411414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411347    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-oauth-serving-cert\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411378    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-service-ca\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411395    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-trusted-ca-bundle\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411414    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-oauth-config\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411497    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-console-config\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411554    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-serving-cert\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411579    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5m65\" (UniqueName: \"kubernetes.io/projected/46419cdf-98d8-4605-9a3e-28f035548c6e-kube-api-access-h5m65\") pod \"46419cdf-98d8-4605-9a3e-28f035548c6e\" (UID: \"46419cdf-98d8-4605-9a3e-28f035548c6e\") "
Sep 16 13:36:38.411866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411780    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:38.411866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.411818    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-service-ca" (OuterVolumeSpecName: "service-ca") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:38.412126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.412077    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:38.412230 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.412151    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-console-config" (OuterVolumeSpecName: "console-config") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:38.414137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.414105    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:36:38.414263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.414236    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46419cdf-98d8-4605-9a3e-28f035548c6e-kube-api-access-h5m65" (OuterVolumeSpecName: "kube-api-access-h5m65") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "kube-api-access-h5m65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:36:38.414327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.414305    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "46419cdf-98d8-4605-9a3e-28f035548c6e" (UID: "46419cdf-98d8-4605-9a3e-28f035548c6e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:36:38.436808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.436779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:38.512222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512194    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-oauth-serving-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.512222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512217    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.512222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512226    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-trusted-ca-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.512410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512235    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-oauth-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.512410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512243    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46419cdf-98d8-4605-9a3e-28f035548c6e-console-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.512410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512252    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46419cdf-98d8-4605-9a3e-28f035548c6e-console-serving-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.512410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.512259    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5m65\" (UniqueName: \"kubernetes.io/projected/46419cdf-98d8-4605-9a3e-28f035548c6e-kube-api-access-h5m65\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:38.541436 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.541399    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:36:38.969876 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.969829    2556 generic.go:356] "Generic (PLEG): container finished" podID="46419cdf-98d8-4605-9a3e-28f035548c6e" containerID="6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962" exitCode=0
Sep 16 13:36:38.970115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.969884    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dc5df7fb-x6tkk"
Sep 16 13:36:38.970115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.969924    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc5df7fb-x6tkk" event={"ID":"46419cdf-98d8-4605-9a3e-28f035548c6e","Type":"ContainerDied","Data":"6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962"}
Sep 16 13:36:38.970115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.969995    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dc5df7fb-x6tkk" event={"ID":"46419cdf-98d8-4605-9a3e-28f035548c6e","Type":"ContainerDied","Data":"f1641b198ae6d80daecd88ed84ae4c67f28384cd4f974614164a78680ee81d17"}
Sep 16 13:36:38.970115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.970017    2556 scope.go:117] "RemoveContainer" containerID="6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962"
Sep 16 13:36:38.984627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.984076    2556 scope.go:117] "RemoveContainer" containerID="6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962"
Sep 16 13:36:38.987401 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:36:38.987374    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962\": container with ID starting with 6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962 not found: ID does not exist" containerID="6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962"
Sep 16 13:36:38.987504 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.987415    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962"} err="failed to get container status \"6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962\": rpc error: code = NotFound desc = could not find container \"6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962\": container with ID starting with 6a4776b37a1de492ca5bf83280432b0f583f298200584ce02c1fbb9998ace962 not found: ID does not exist"
Sep 16 13:36:38.992012 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.991991    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dc5df7fb-x6tkk"]
Sep 16 13:36:38.995785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:38.995764    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8dc5df7fb-x6tkk"]
Sep 16 13:36:40.057630 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:40.057597    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46419cdf-98d8-4605-9a3e-28f035548c6e" path="/var/lib/kubelet/pods/46419cdf-98d8-4605-9a3e-28f035548c6e/volumes"
Sep 16 13:36:48.974811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:48.974747    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75dbcfdf6c-mwsn6" podUID="8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" containerName="console" containerID="cri-o://c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4" gracePeriod=15
Sep 16 13:36:49.221234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.221211    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:36:49.296104 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296015    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c584\" (UniqueName: \"kubernetes.io/projected/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-kube-api-access-6c584\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296104 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296060    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-oauth-serving-cert\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296083    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-config\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296131    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-service-ca\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296156    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-trusted-ca-bundle\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296193    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-oauth-config\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296258    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-serving-cert\") pod \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\" (UID: \"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce\") "
Sep 16 13:36:49.296563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296536    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:49.296563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296546    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-config" (OuterVolumeSpecName: "console-config") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:49.296675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296572    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:49.296675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.296619    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:36:49.298458 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.298435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-kube-api-access-6c584" (OuterVolumeSpecName: "kube-api-access-6c584") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "kube-api-access-6c584". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:36:49.298569 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.298524    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:36:49.298569 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.298543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" (UID: "8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:36:49.397560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397526    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c584\" (UniqueName: \"kubernetes.io/projected/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-kube-api-access-6c584\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:49.397560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397552    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-oauth-serving-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:49.397560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397565    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:49.397755 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397577    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:49.397755 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397588    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-trusted-ca-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:49.397755 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397599    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-oauth-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:49.397755 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:49.397610    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce-console-serving-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:36:50.017474 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.017441    2556 generic.go:356] "Generic (PLEG): container finished" podID="8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" containerID="c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4" exitCode=0
Sep 16 13:36:50.017881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.017484    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dbcfdf6c-mwsn6" event={"ID":"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce","Type":"ContainerDied","Data":"c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4"}
Sep 16 13:36:50.017881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.017488    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75dbcfdf6c-mwsn6"
Sep 16 13:36:50.017881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.017517    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75dbcfdf6c-mwsn6" event={"ID":"8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce","Type":"ContainerDied","Data":"8026e43a7373459e3054d84ba0196bcef05faf16ca03b88e19831bf923584faf"}
Sep 16 13:36:50.017881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.017536    2556 scope.go:117] "RemoveContainer" containerID="c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4"
Sep 16 13:36:50.026521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.026503    2556 scope.go:117] "RemoveContainer" containerID="c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4"
Sep 16 13:36:50.026793 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:36:50.026775    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4\": container with ID starting with c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4 not found: ID does not exist" containerID="c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4"
Sep 16 13:36:50.026830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.026803    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4"} err="failed to get container status \"c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4\": rpc error: code = NotFound desc = could not find container \"c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4\": container with ID starting with c164380f0a798f79d4027119af40eedd6f34d90727291790564bd3590a0e46f4 not found: ID does not exist"
Sep 16 13:36:50.038756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.038732    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75dbcfdf6c-mwsn6"]
Sep 16 13:36:50.040141 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.040119    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75dbcfdf6c-mwsn6"]
Sep 16 13:36:50.057720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:36:50.057696    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" path="/var/lib/kubelet/pods/8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce/volumes"
Sep 16 13:37:06.759960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:06.759919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:18.105873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.105841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:18.205200 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.205163    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:18.714688 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.714656    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"]
Sep 16 13:37:18.714950 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.714939    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46419cdf-98d8-4605-9a3e-28f035548c6e" containerName="console"
Sep 16 13:37:18.714991 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.714953    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="46419cdf-98d8-4605-9a3e-28f035548c6e" containerName="console"
Sep 16 13:37:18.714991 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.714975    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" containerName="console"
Sep 16 13:37:18.714991 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.714981    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" containerName="console"
Sep 16 13:37:18.715077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.715026    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca0c8e0-28da-4a56-b4ba-d53160b8c9ce" containerName="console"
Sep 16 13:37:18.715077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.715033    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="46419cdf-98d8-4605-9a3e-28f035548c6e" containerName="console"
Sep 16 13:37:18.719322 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.719307    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.721174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.721154    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 13:37:18.721278 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.721224    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 13:37:18.721775 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.721761    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2mrld\""
Sep 16 13:37:18.727240 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.727221    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"]
Sep 16 13:37:18.728346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.728325    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.728451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.728375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.728451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.728415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j9nz\" (UniqueName: \"kubernetes.io/projected/ddc9c449-7027-48a8-a036-3d9a7c06717a-kube-api-access-6j9nz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.828700 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.828671    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.828824 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.828711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j9nz\" (UniqueName: \"kubernetes.io/projected/ddc9c449-7027-48a8-a036-3d9a7c06717a-kube-api-access-6j9nz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.828824 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.828745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.829035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.829017    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.829081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.829065    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:18.841281 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:18.841258    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j9nz\" (UniqueName: \"kubernetes.io/projected/ddc9c449-7027-48a8-a036-3d9a7c06717a-kube-api-access-6j9nz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:19.029124 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:19.029013    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:19.296042 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:19.295929    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"]
Sep 16 13:37:19.298231 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:37:19.298205    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc9c449_7027_48a8_a036_3d9a7c06717a.slice/crio-52eee0bc102c24568cb422daf5e34fb958d850219d50f71da0a44c17fda81db0 WatchSource:0}: Error finding container 52eee0bc102c24568cb422daf5e34fb958d850219d50f71da0a44c17fda81db0: Status 404 returned error can't find the container with id 52eee0bc102c24568cb422daf5e34fb958d850219d50f71da0a44c17fda81db0
Sep 16 13:37:20.128490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:20.128452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" event={"ID":"ddc9c449-7027-48a8-a036-3d9a7c06717a","Type":"ContainerStarted","Data":"52eee0bc102c24568cb422daf5e34fb958d850219d50f71da0a44c17fda81db0"}
Sep 16 13:37:27.156883 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:27.156845    2556 generic.go:356] "Generic (PLEG): container finished" podID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerID="e8a2ee8570e60bffd25342c47281f95e463820a7bd8922759ec92056ba7b235e" exitCode=0
Sep 16 13:37:27.157268 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:27.156918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" event={"ID":"ddc9c449-7027-48a8-a036-3d9a7c06717a","Type":"ContainerDied","Data":"e8a2ee8570e60bffd25342c47281f95e463820a7bd8922759ec92056ba7b235e"}
Sep 16 13:37:30.170619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:30.170584    2556 generic.go:356] "Generic (PLEG): container finished" podID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerID="2ba21581ca54dc455d69b806495accd746325663d4e42984b00e77a02b8fe474" exitCode=0
Sep 16 13:37:30.170971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:30.170637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" event={"ID":"ddc9c449-7027-48a8-a036-3d9a7c06717a","Type":"ContainerDied","Data":"2ba21581ca54dc455d69b806495accd746325663d4e42984b00e77a02b8fe474"}
Sep 16 13:37:37.205692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:37.205660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" event={"ID":"ddc9c449-7027-48a8-a036-3d9a7c06717a","Type":"ContainerStarted","Data":"cf974f88f1fe8eaf5d3839fba36e517b88b2681ca5bf5308971a505484aacd75"}
Sep 16 13:37:37.221742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:37.221688    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" podStartSLOduration=3.060977927 podStartE2EDuration="19.221665937s" totalImagesPullingTime="16.16068801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:37.219948613 +0000 UTC m=+267.685885524" watchObservedRunningTime="2026-09-16 13:37:37.221665937 +0000 UTC m=+267.687602844"
Sep 16 13:37:38.211271 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:38.211223    2556 generic.go:356] "Generic (PLEG): container finished" podID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerID="cf974f88f1fe8eaf5d3839fba36e517b88b2681ca5bf5308971a505484aacd75" exitCode=0
Sep 16 13:37:38.211271 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:38.211253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" event={"ID":"ddc9c449-7027-48a8-a036-3d9a7c06717a","Type":"ContainerDied","Data":"cf974f88f1fe8eaf5d3839fba36e517b88b2681ca5bf5308971a505484aacd75"}
Sep 16 13:37:39.339906 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.339881    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:39.406163 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.406135    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j9nz\" (UniqueName: \"kubernetes.io/projected/ddc9c449-7027-48a8-a036-3d9a7c06717a-kube-api-access-6j9nz\") pod \"ddc9c449-7027-48a8-a036-3d9a7c06717a\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") "
Sep 16 13:37:39.406301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.406195    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-util\") pod \"ddc9c449-7027-48a8-a036-3d9a7c06717a\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") "
Sep 16 13:37:39.406301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.406222    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-bundle\") pod \"ddc9c449-7027-48a8-a036-3d9a7c06717a\" (UID: \"ddc9c449-7027-48a8-a036-3d9a7c06717a\") "
Sep 16 13:37:39.406856 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.406835    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-bundle" (OuterVolumeSpecName: "bundle") pod "ddc9c449-7027-48a8-a036-3d9a7c06717a" (UID: "ddc9c449-7027-48a8-a036-3d9a7c06717a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:37:39.408444 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.408424    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc9c449-7027-48a8-a036-3d9a7c06717a-kube-api-access-6j9nz" (OuterVolumeSpecName: "kube-api-access-6j9nz") pod "ddc9c449-7027-48a8-a036-3d9a7c06717a" (UID: "ddc9c449-7027-48a8-a036-3d9a7c06717a"). InnerVolumeSpecName "kube-api-access-6j9nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:37:39.410210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.410188    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-util" (OuterVolumeSpecName: "util") pod "ddc9c449-7027-48a8-a036-3d9a7c06717a" (UID: "ddc9c449-7027-48a8-a036-3d9a7c06717a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:37:39.506932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.506876    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j9nz\" (UniqueName: \"kubernetes.io/projected/ddc9c449-7027-48a8-a036-3d9a7c06717a-kube-api-access-6j9nz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:37:39.506932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.506897    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:37:39.506932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:39.506907    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ddc9c449-7027-48a8-a036-3d9a7c06717a-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:37:40.220499 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:40.220467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g" event={"ID":"ddc9c449-7027-48a8-a036-3d9a7c06717a","Type":"ContainerDied","Data":"52eee0bc102c24568cb422daf5e34fb958d850219d50f71da0a44c17fda81db0"}
Sep 16 13:37:40.220660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:40.220511    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406jt7g"
Sep 16 13:37:40.220660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:40.220520    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52eee0bc102c24568cb422daf5e34fb958d850219d50f71da0a44c17fda81db0"
Sep 16 13:37:44.737508 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:44.737475    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:45.567384 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:45.567352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:46.722697 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.722666    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"]
Sep 16 13:37:46.723170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723129    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="pull"
Sep 16 13:37:46.723170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723146    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="pull"
Sep 16 13:37:46.723170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723158    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="extract"
Sep 16 13:37:46.723170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723165    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="extract"
Sep 16 13:37:46.723373 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723183    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="util"
Sep 16 13:37:46.723373 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723192    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="util"
Sep 16 13:37:46.723373 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.723302    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc9c449-7027-48a8-a036-3d9a7c06717a" containerName="extract"
Sep 16 13:37:46.726670 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.726649    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:46.728943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.728918    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vzdj6\""
Sep 16 13:37:46.728943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.728943    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 13:37:46.729131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.728981    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 13:37:46.740144 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.740124    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"]
Sep 16 13:37:46.867685 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.867645    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16bf07b5-cdf4-42dc-8001-e6688be87c41-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s66dj\" (UID: \"16bf07b5-cdf4-42dc-8001-e6688be87c41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:46.867851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.867695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7d64\" (UniqueName: \"kubernetes.io/projected/16bf07b5-cdf4-42dc-8001-e6688be87c41-kube-api-access-g7d64\") pod \"cert-manager-operator-controller-manager-55c444bf86-s66dj\" (UID: \"16bf07b5-cdf4-42dc-8001-e6688be87c41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:46.968852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.968815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16bf07b5-cdf4-42dc-8001-e6688be87c41-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s66dj\" (UID: \"16bf07b5-cdf4-42dc-8001-e6688be87c41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:46.969158 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.969129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7d64\" (UniqueName: \"kubernetes.io/projected/16bf07b5-cdf4-42dc-8001-e6688be87c41-kube-api-access-g7d64\") pod \"cert-manager-operator-controller-manager-55c444bf86-s66dj\" (UID: \"16bf07b5-cdf4-42dc-8001-e6688be87c41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:46.969435 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.969414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16bf07b5-cdf4-42dc-8001-e6688be87c41-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-s66dj\" (UID: \"16bf07b5-cdf4-42dc-8001-e6688be87c41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:46.978137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:46.978064    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7d64\" (UniqueName: \"kubernetes.io/projected/16bf07b5-cdf4-42dc-8001-e6688be87c41-kube-api-access-g7d64\") pod \"cert-manager-operator-controller-manager-55c444bf86-s66dj\" (UID: \"16bf07b5-cdf4-42dc-8001-e6688be87c41\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:47.036007 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:47.035983    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"
Sep 16 13:37:47.306422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:47.306398    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj"]
Sep 16 13:37:47.309042 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:37:47.309009    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bf07b5_cdf4_42dc_8001_e6688be87c41.slice/crio-cb64caa4547af13e084984de40b454764afcbf5180bc44623f634bdbc7b0d500 WatchSource:0}: Error finding container cb64caa4547af13e084984de40b454764afcbf5180bc44623f634bdbc7b0d500: Status 404 returned error can't find the container with id cb64caa4547af13e084984de40b454764afcbf5180bc44623f634bdbc7b0d500
Sep 16 13:37:48.252836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:48.252797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj" event={"ID":"16bf07b5-cdf4-42dc-8001-e6688be87c41","Type":"ContainerStarted","Data":"cb64caa4547af13e084984de40b454764afcbf5180bc44623f634bdbc7b0d500"}
Sep 16 13:37:48.343386 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:48.343352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:48.445872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:48.445836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:49.259358 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:49.259325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj" event={"ID":"16bf07b5-cdf4-42dc-8001-e6688be87c41","Type":"ContainerStarted","Data":"d62898b9764fd9d5c356989c22df1d7ec4cf3b9ecb4978ff6aee0265b1f1d951"}
Sep 16 13:37:49.283221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:49.283176    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-s66dj" podStartSLOduration=1.559019931 podStartE2EDuration="3.283160009s" totalImagesPullingTime="1.724140078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:49.280834782 +0000 UTC m=+279.746771708" watchObservedRunningTime="2026-09-16 13:37:49.283160009 +0000 UTC m=+279.749096915"
Sep 16 13:37:49.648065 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:49.648033    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:49.753177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:49.753137    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:50.038616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.038584    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:50.137367 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.137318    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:50.390548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.390457    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"]
Sep 16 13:37:50.392749 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.392731    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.394623 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.394604    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 13:37:50.394825 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.394812    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 13:37:50.395383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.395355    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2mrld\""
Sep 16 13:37:50.402223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.402202    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"]
Sep 16 13:37:50.498026 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.497996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6pt\" (UniqueName: \"kubernetes.io/projected/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-kube-api-access-vw6pt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.498195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.498043    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.498195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.498064    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.599183 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.599148    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.599321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.599188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.599321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.599247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6pt\" (UniqueName: \"kubernetes.io/projected/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-kube-api-access-vw6pt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.599540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.599520    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.599642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.599545    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.606833 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.606813    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6pt\" (UniqueName: \"kubernetes.io/projected/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-kube-api-access-vw6pt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.703187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.703163    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:50.968651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.968627    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"]
Sep 16 13:37:50.971032 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:37:50.971005    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3420ff8_74f8_4d36_addc_14a5e39e6a1a.slice/crio-81b06aff56112dd47309685beb07f75d363eda028f22d3bb971dbc80ebe89a02 WatchSource:0}: Error finding container 81b06aff56112dd47309685beb07f75d363eda028f22d3bb971dbc80ebe89a02: Status 404 returned error can't find the container with id 81b06aff56112dd47309685beb07f75d363eda028f22d3bb971dbc80ebe89a02
Sep 16 13:37:50.985690 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.985666    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4nh5c"]
Sep 16 13:37:50.988844 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.988825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:50.991043 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.990755    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 13:37:50.991043 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.990934    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 13:37:50.991043 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.991005    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-rqh25\""
Sep 16 13:37:50.997647 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:50.997616    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4nh5c"]
Sep 16 13:37:51.104615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.104579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxn5v\" (UniqueName: \"kubernetes.io/projected/7d66c7e1-65cc-4d87-9009-75902c23ce87-kube-api-access-xxn5v\") pod \"cert-manager-webhook-6486ffd997-4nh5c\" (UID: \"7d66c7e1-65cc-4d87-9009-75902c23ce87\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.104748 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.104670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d66c7e1-65cc-4d87-9009-75902c23ce87-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4nh5c\" (UID: \"7d66c7e1-65cc-4d87-9009-75902c23ce87\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.205412 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.205381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxn5v\" (UniqueName: \"kubernetes.io/projected/7d66c7e1-65cc-4d87-9009-75902c23ce87-kube-api-access-xxn5v\") pod \"cert-manager-webhook-6486ffd997-4nh5c\" (UID: \"7d66c7e1-65cc-4d87-9009-75902c23ce87\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.205564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.205443    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d66c7e1-65cc-4d87-9009-75902c23ce87-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4nh5c\" (UID: \"7d66c7e1-65cc-4d87-9009-75902c23ce87\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.213398 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.213373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7d66c7e1-65cc-4d87-9009-75902c23ce87-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4nh5c\" (UID: \"7d66c7e1-65cc-4d87-9009-75902c23ce87\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.213603 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.213584    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxn5v\" (UniqueName: \"kubernetes.io/projected/7d66c7e1-65cc-4d87-9009-75902c23ce87-kube-api-access-xxn5v\") pod \"cert-manager-webhook-6486ffd997-4nh5c\" (UID: \"7d66c7e1-65cc-4d87-9009-75902c23ce87\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.269519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.269454    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerID="375afd8913abc4f7569b4b7d44642831b299a7c06c7e6093f45e645262871e8d" exitCode=0
Sep 16 13:37:51.269611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.269525    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc" event={"ID":"e3420ff8-74f8-4d36-addc-14a5e39e6a1a","Type":"ContainerDied","Data":"375afd8913abc4f7569b4b7d44642831b299a7c06c7e6093f45e645262871e8d"}
Sep 16 13:37:51.269611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.269601    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc" event={"ID":"e3420ff8-74f8-4d36-addc-14a5e39e6a1a","Type":"ContainerStarted","Data":"81b06aff56112dd47309685beb07f75d363eda028f22d3bb971dbc80ebe89a02"}
Sep 16 13:37:51.348395 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.348371    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:51.615883 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:51.615857    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4nh5c"]
Sep 16 13:37:51.618365 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:37:51.618334    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d66c7e1_65cc_4d87_9009_75902c23ce87.slice/crio-4264e05791bc63a2641c1e766cc61d91353219036fd752ade4d5671b3d5871b7 WatchSource:0}: Error finding container 4264e05791bc63a2641c1e766cc61d91353219036fd752ade4d5671b3d5871b7: Status 404 returned error can't find the container with id 4264e05791bc63a2641c1e766cc61d91353219036fd752ade4d5671b3d5871b7
Sep 16 13:37:52.276637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:52.276595    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c" event={"ID":"7d66c7e1-65cc-4d87-9009-75902c23ce87","Type":"ContainerStarted","Data":"4264e05791bc63a2641c1e766cc61d91353219036fd752ade4d5671b3d5871b7"}
Sep 16 13:37:52.417274 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:52.417232    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:52.824527 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:52.824468    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:53.284122 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:53.284036    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc" event={"ID":"e3420ff8-74f8-4d36-addc-14a5e39e6a1a","Type":"ContainerStarted","Data":"25ab2042e8e2dc070e4bc9fa351b9506d6af0888d1cea2b1bf928c4c58ed033f"}
Sep 16 13:37:54.293891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:54.293850    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerID="25ab2042e8e2dc070e4bc9fa351b9506d6af0888d1cea2b1bf928c4c58ed033f" exitCode=0
Sep 16 13:37:54.294340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:54.293907    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc" event={"ID":"e3420ff8-74f8-4d36-addc-14a5e39e6a1a","Type":"ContainerDied","Data":"25ab2042e8e2dc070e4bc9fa351b9506d6af0888d1cea2b1bf928c4c58ed033f"}
Sep 16 13:37:55.146019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.145981    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:55.280928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.280892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:55.300183 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.300151    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c" event={"ID":"7d66c7e1-65cc-4d87-9009-75902c23ce87","Type":"ContainerStarted","Data":"459102fa830a4eaa241be8c5351eca9845b444f6593a37a3827690278034ec4a"}
Sep 16 13:37:55.300572 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.300291    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:37:55.302299 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.302277    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerID="22292f192818cd455dbcd70f1efb33999e7727283bb3b6f24bc931ace4c50733" exitCode=0
Sep 16 13:37:55.302399 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.302340    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc" event={"ID":"e3420ff8-74f8-4d36-addc-14a5e39e6a1a","Type":"ContainerDied","Data":"22292f192818cd455dbcd70f1efb33999e7727283bb3b6f24bc931ace4c50733"}
Sep 16 13:37:55.316523 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:55.316471    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c" podStartSLOduration=2.562996658 podStartE2EDuration="5.316456897s" totalImagesPullingTime="2.753460239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:37:55.314660109 +0000 UTC m=+285.780597015" watchObservedRunningTime="2026-09-16 13:37:55.316456897 +0000 UTC m=+285.782393803"
Sep 16 13:37:56.424870 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.424842    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:56.551501 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.551470    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw6pt\" (UniqueName: \"kubernetes.io/projected/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-kube-api-access-vw6pt\") pod \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") "
Sep 16 13:37:56.551501 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.551502    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-bundle\") pod \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") "
Sep 16 13:37:56.551730 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.551533    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-util\") pod \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\" (UID: \"e3420ff8-74f8-4d36-addc-14a5e39e6a1a\") "
Sep 16 13:37:56.551928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.551898    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-bundle" (OuterVolumeSpecName: "bundle") pod "e3420ff8-74f8-4d36-addc-14a5e39e6a1a" (UID: "e3420ff8-74f8-4d36-addc-14a5e39e6a1a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:37:56.553738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.553711    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-kube-api-access-vw6pt" (OuterVolumeSpecName: "kube-api-access-vw6pt") pod "e3420ff8-74f8-4d36-addc-14a5e39e6a1a" (UID: "e3420ff8-74f8-4d36-addc-14a5e39e6a1a"). InnerVolumeSpecName "kube-api-access-vw6pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:37:56.582779 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.582753    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-util" (OuterVolumeSpecName: "util") pod "e3420ff8-74f8-4d36-addc-14a5e39e6a1a" (UID: "e3420ff8-74f8-4d36-addc-14a5e39e6a1a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:37:56.652399 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.652383    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:37:56.652473 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.652402    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6pt\" (UniqueName: \"kubernetes.io/projected/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-kube-api-access-vw6pt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:37:56.652473 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:56.652412    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e3420ff8-74f8-4d36-addc-14a5e39e6a1a-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:37:57.312469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:57.312440    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc"
Sep 16 13:37:57.312469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:57.312439    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqf8xc" event={"ID":"e3420ff8-74f8-4d36-addc-14a5e39e6a1a","Type":"ContainerDied","Data":"81b06aff56112dd47309685beb07f75d363eda028f22d3bb971dbc80ebe89a02"}
Sep 16 13:37:57.312657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:57.312523    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81b06aff56112dd47309685beb07f75d363eda028f22d3bb971dbc80ebe89a02"
Sep 16 13:37:59.675271 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:59.675240    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:37:59.775992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:37:59.775950    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:00.629195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.629161    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:00.771103 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771068    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rd2mq"]
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771365    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="extract"
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771378    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="extract"
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771388    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="pull"
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771395    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="pull"
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771403    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="util"
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771408    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="util"
Sep 16 13:38:00.771456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.771459    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3420ff8-74f8-4d36-addc-14a5e39e6a1a" containerName="extract"
Sep 16 13:38:00.775434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.775419    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:00.777460 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.777431    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kcvts\""
Sep 16 13:38:00.782467 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.782236    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rd2mq"]
Sep 16 13:38:00.889287 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.889190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6640b0d0-d869-4b8d-83b5-dc2198d6fd82-bound-sa-token\") pod \"cert-manager-55b4d954d5-rd2mq\" (UID: \"6640b0d0-d869-4b8d-83b5-dc2198d6fd82\") " pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:00.889287 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.889252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwrk\" (UniqueName: \"kubernetes.io/projected/6640b0d0-d869-4b8d-83b5-dc2198d6fd82-kube-api-access-znwrk\") pod \"cert-manager-55b4d954d5-rd2mq\" (UID: \"6640b0d0-d869-4b8d-83b5-dc2198d6fd82\") " pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:00.990714 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.990675    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwrk\" (UniqueName: \"kubernetes.io/projected/6640b0d0-d869-4b8d-83b5-dc2198d6fd82-kube-api-access-znwrk\") pod \"cert-manager-55b4d954d5-rd2mq\" (UID: \"6640b0d0-d869-4b8d-83b5-dc2198d6fd82\") " pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:00.990906 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.990739    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6640b0d0-d869-4b8d-83b5-dc2198d6fd82-bound-sa-token\") pod \"cert-manager-55b4d954d5-rd2mq\" (UID: \"6640b0d0-d869-4b8d-83b5-dc2198d6fd82\") " pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:00.998760 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.998730    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6640b0d0-d869-4b8d-83b5-dc2198d6fd82-bound-sa-token\") pod \"cert-manager-55b4d954d5-rd2mq\" (UID: \"6640b0d0-d869-4b8d-83b5-dc2198d6fd82\") " pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:00.998859 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:00.998803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwrk\" (UniqueName: \"kubernetes.io/projected/6640b0d0-d869-4b8d-83b5-dc2198d6fd82-kube-api-access-znwrk\") pod \"cert-manager-55b4d954d5-rd2mq\" (UID: \"6640b0d0-d869-4b8d-83b5-dc2198d6fd82\") " pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:01.084569 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:01.084537    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-rd2mq"
Sep 16 13:38:01.161556 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:01.161483    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:01.262797 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:01.262756    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:01.308988 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:01.308959    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4nh5c"
Sep 16 13:38:01.357485 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:01.357462    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-rd2mq"]
Sep 16 13:38:01.360146 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:38:01.360114    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6640b0d0_d869_4b8d_83b5_dc2198d6fd82.slice/crio-4286fc158baf6f4d985f550b4db7ad231715ea43bf5ba8b54b907a94e1d24eb9 WatchSource:0}: Error finding container 4286fc158baf6f4d985f550b4db7ad231715ea43bf5ba8b54b907a94e1d24eb9: Status 404 returned error can't find the container with id 4286fc158baf6f4d985f550b4db7ad231715ea43bf5ba8b54b907a94e1d24eb9
Sep 16 13:38:02.335472 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:02.335442    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rd2mq" event={"ID":"6640b0d0-d869-4b8d-83b5-dc2198d6fd82","Type":"ContainerStarted","Data":"e50fde21f02812a8bc909f9bb6ff8c72210b012662663d5e3d8b6332b54adf81"}
Sep 16 13:38:02.335831 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:02.335483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-rd2mq" event={"ID":"6640b0d0-d869-4b8d-83b5-dc2198d6fd82","Type":"ContainerStarted","Data":"4286fc158baf6f4d985f550b4db7ad231715ea43bf5ba8b54b907a94e1d24eb9"}
Sep 16 13:38:02.352473 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:02.352417    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-rd2mq" podStartSLOduration=2.352400057 podStartE2EDuration="2.352400057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:02.351677353 +0000 UTC m=+292.817614259" watchObservedRunningTime="2026-09-16 13:38:02.352400057 +0000 UTC m=+292.818336963"
Sep 16 13:38:07.759347 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:07.759313    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:07.814929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:07.814896    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:09.362841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:09.362806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:09.970081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:09.970054    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 13:38:10.148628 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.148488    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"]
Sep 16 13:38:10.219115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.178824    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"]
Sep 16 13:38:10.219115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.178920    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.219115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.181418    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 13:38:10.219115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.181446    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2mrld\""
Sep 16 13:38:10.219115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.182004    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 13:38:10.272992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.272905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.272992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.272948    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klk5k\" (UniqueName: \"kubernetes.io/projected/3b2c52b1-0bb5-45db-a989-7875f86ba722-kube-api-access-klk5k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.272992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.272989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.374161 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.374130    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.374560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.374220    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.374560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.374252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klk5k\" (UniqueName: \"kubernetes.io/projected/3b2c52b1-0bb5-45db-a989-7875f86ba722-kube-api-access-klk5k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.374560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.374503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.374680 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.374562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.381592 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.381568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klk5k\" (UniqueName: \"kubernetes.io/projected/3b2c52b1-0bb5-45db-a989-7875f86ba722-kube-api-access-klk5k\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.487787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.487753    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:10.760239 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:10.760198    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"]
Sep 16 13:38:10.762636 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:38:10.762606    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2c52b1_0bb5_45db_a989_7875f86ba722.slice/crio-71ed19bbde6017547fc6b9eb60fc839b9b921735abcff220e71d8472515dba47 WatchSource:0}: Error finding container 71ed19bbde6017547fc6b9eb60fc839b9b921735abcff220e71d8472515dba47: Status 404 returned error can't find the container with id 71ed19bbde6017547fc6b9eb60fc839b9b921735abcff220e71d8472515dba47
Sep 16 13:38:11.376037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:11.376005    2556 generic.go:356] "Generic (PLEG): container finished" podID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerID="4339c0b17bf61cf638828890d9275b07f8fc8a8a0519c357fe5363f0d723ad9f" exitCode=0
Sep 16 13:38:11.376434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:11.376105    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw" event={"ID":"3b2c52b1-0bb5-45db-a989-7875f86ba722","Type":"ContainerDied","Data":"4339c0b17bf61cf638828890d9275b07f8fc8a8a0519c357fe5363f0d723ad9f"}
Sep 16 13:38:11.376434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:11.376152    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw" event={"ID":"3b2c52b1-0bb5-45db-a989-7875f86ba722","Type":"ContainerStarted","Data":"71ed19bbde6017547fc6b9eb60fc839b9b921735abcff220e71d8472515dba47"}
Sep 16 13:38:11.376589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:11.376574    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:38:12.385171 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:12.385133    2556 generic.go:356] "Generic (PLEG): container finished" podID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerID="b3b029e00dcf2db68e5b5a2b6e4bf66356690f0d1872f6a5f432030c8fef215c" exitCode=0
Sep 16 13:38:12.385568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:12.385188    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw" event={"ID":"3b2c52b1-0bb5-45db-a989-7875f86ba722","Type":"ContainerDied","Data":"b3b029e00dcf2db68e5b5a2b6e4bf66356690f0d1872f6a5f432030c8fef215c"}
Sep 16 13:38:13.391905 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:13.391868    2556 generic.go:356] "Generic (PLEG): container finished" podID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerID="427b0c047320b0a7c3aea06952903f296b39a927c5bfb4826840d987e97980d1" exitCode=0
Sep 16 13:38:13.392311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:13.391946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw" event={"ID":"3b2c52b1-0bb5-45db-a989-7875f86ba722","Type":"ContainerDied","Data":"427b0c047320b0a7c3aea06952903f296b39a927c5bfb4826840d987e97980d1"}
Sep 16 13:38:14.517677 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.517653    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:14.610035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.609995    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klk5k\" (UniqueName: \"kubernetes.io/projected/3b2c52b1-0bb5-45db-a989-7875f86ba722-kube-api-access-klk5k\") pod \"3b2c52b1-0bb5-45db-a989-7875f86ba722\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") "
Sep 16 13:38:14.610035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.610041    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-bundle\") pod \"3b2c52b1-0bb5-45db-a989-7875f86ba722\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") "
Sep 16 13:38:14.610288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.610129    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-util\") pod \"3b2c52b1-0bb5-45db-a989-7875f86ba722\" (UID: \"3b2c52b1-0bb5-45db-a989-7875f86ba722\") "
Sep 16 13:38:14.610808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.610788    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-bundle" (OuterVolumeSpecName: "bundle") pod "3b2c52b1-0bb5-45db-a989-7875f86ba722" (UID: "3b2c52b1-0bb5-45db-a989-7875f86ba722"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:14.612403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.612378    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b2c52b1-0bb5-45db-a989-7875f86ba722-kube-api-access-klk5k" (OuterVolumeSpecName: "kube-api-access-klk5k") pod "3b2c52b1-0bb5-45db-a989-7875f86ba722" (UID: "3b2c52b1-0bb5-45db-a989-7875f86ba722"). InnerVolumeSpecName "kube-api-access-klk5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:38:14.618312 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.618282    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-util" (OuterVolumeSpecName: "util") pod "3b2c52b1-0bb5-45db-a989-7875f86ba722" (UID: "3b2c52b1-0bb5-45db-a989-7875f86ba722"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:14.711415 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.711366    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klk5k\" (UniqueName: \"kubernetes.io/projected/3b2c52b1-0bb5-45db-a989-7875f86ba722-kube-api-access-klk5k\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:38:14.711415 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.711400    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:38:14.711415 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:14.711409    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b2c52b1-0bb5-45db-a989-7875f86ba722-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:38:15.408378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:15.408345    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw"
Sep 16 13:38:15.408546 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:15.408358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af4b7bw" event={"ID":"3b2c52b1-0bb5-45db-a989-7875f86ba722","Type":"ContainerDied","Data":"71ed19bbde6017547fc6b9eb60fc839b9b921735abcff220e71d8472515dba47"}
Sep 16 13:38:15.408546 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:15.408460    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71ed19bbde6017547fc6b9eb60fc839b9b921735abcff220e71d8472515dba47"
Sep 16 13:38:19.972307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972268    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"]
Sep 16 13:38:19.972761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972716    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="extract"
Sep 16 13:38:19.972761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972735    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="extract"
Sep 16 13:38:19.972761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972756    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="pull"
Sep 16 13:38:19.972916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972764    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="pull"
Sep 16 13:38:19.972916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972776    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="util"
Sep 16 13:38:19.972916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972784    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="util"
Sep 16 13:38:19.972916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.972856    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b2c52b1-0bb5-45db-a989-7875f86ba722" containerName="extract"
Sep 16 13:38:19.975000 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.974978    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:19.977261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.977241    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 13:38:19.977814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.977797    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2mrld\""
Sep 16 13:38:19.977898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.977812    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 13:38:19.984130 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:19.984110    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"]
Sep 16 13:38:20.055531 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.055501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.055692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.055537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.055692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.055604    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnnt\" (UniqueName: \"kubernetes.io/projected/2c77ae70-53da-4276-b67e-588b00283b0a-kube-api-access-fnnnt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.156226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.156188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.156226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.156227    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.156444 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.156342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnnt\" (UniqueName: \"kubernetes.io/projected/2c77ae70-53da-4276-b67e-588b00283b0a-kube-api-access-fnnnt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.156629 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.156610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.156665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.156618    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.164552 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.164526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnnt\" (UniqueName: \"kubernetes.io/projected/2c77ae70-53da-4276-b67e-588b00283b0a-kube-api-access-fnnnt\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.284798 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.284709    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:20.569081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:20.569051    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"]
Sep 16 13:38:20.571245 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:38:20.571204    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c77ae70_53da_4276_b67e_588b00283b0a.slice/crio-a2a8bf209f73be90cb18a5480d733e6e8bcd8b6554e479cfef8e18af20eba668 WatchSource:0}: Error finding container a2a8bf209f73be90cb18a5480d733e6e8bcd8b6554e479cfef8e18af20eba668: Status 404 returned error can't find the container with id a2a8bf209f73be90cb18a5480d733e6e8bcd8b6554e479cfef8e18af20eba668
Sep 16 13:38:21.052114 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:21.052061    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:21.441989 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:21.441956    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c77ae70-53da-4276-b67e-588b00283b0a" containerID="3204d0939b2057f0ffc7669badbcbb9c4626cc280ef7402df8f3959df87b2c8e" exitCode=0
Sep 16 13:38:21.442204 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:21.442034    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7" event={"ID":"2c77ae70-53da-4276-b67e-588b00283b0a","Type":"ContainerDied","Data":"3204d0939b2057f0ffc7669badbcbb9c4626cc280ef7402df8f3959df87b2c8e"}
Sep 16 13:38:21.442204 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:21.442073    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7" event={"ID":"2c77ae70-53da-4276-b67e-588b00283b0a","Type":"ContainerStarted","Data":"a2a8bf209f73be90cb18a5480d733e6e8bcd8b6554e479cfef8e18af20eba668"}
Sep 16 13:38:22.130758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.130710    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"]
Sep 16 13:38:22.133281 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.133263    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.135646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.135516    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 13:38:22.135646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.135570    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 13:38:22.135646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.135637    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 13:38:22.135884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.135658    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8t96b\""
Sep 16 13:38:22.135884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.135700    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 13:38:22.147015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.146966    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"]
Sep 16 13:38:22.176145 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.176114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ebe934fe-558a-4c8a-82c0-339af02098f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.176306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.176190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ebe934fe-558a-4c8a-82c0-339af02098f8-manifests\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.176306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.176239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebe934fe-558a-4c8a-82c0-339af02098f8-tmp\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.176306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.176263    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88rg\" (UniqueName: \"kubernetes.io/projected/ebe934fe-558a-4c8a-82c0-339af02098f8-kube-api-access-x88rg\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.176473 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.176343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebe934fe-558a-4c8a-82c0-339af02098f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.277628 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.277589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ebe934fe-558a-4c8a-82c0-339af02098f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.277814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.277648    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ebe934fe-558a-4c8a-82c0-339af02098f8-manifests\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.277814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.277692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebe934fe-558a-4c8a-82c0-339af02098f8-tmp\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.277814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.277717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88rg\" (UniqueName: \"kubernetes.io/projected/ebe934fe-558a-4c8a-82c0-339af02098f8-kube-api-access-x88rg\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.277814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.277750    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebe934fe-558a-4c8a-82c0-339af02098f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.278263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.278234    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ebe934fe-558a-4c8a-82c0-339af02098f8-manifests\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.280755 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.280727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ebe934fe-558a-4c8a-82c0-339af02098f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.280997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.280976    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebe934fe-558a-4c8a-82c0-339af02098f8-tmp\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.281357 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.281335    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ebe934fe-558a-4c8a-82c0-339af02098f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.285872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.285848    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88rg\" (UniqueName: \"kubernetes.io/projected/ebe934fe-558a-4c8a-82c0-339af02098f8-kube-api-access-x88rg\") pod \"opendatahub-operator-controller-manager-6444576fdb-54p72\" (UID: \"ebe934fe-558a-4c8a-82c0-339af02098f8\") " pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.303618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.303588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:22.398602 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.398517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:22.443237 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.443207    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:22.449461 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.449434    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c77ae70-53da-4276-b67e-588b00283b0a" containerID="9d02c3a2b3339a743feab3d591c336fe1d05df319730b9c9fe2a39c0cdea5a29" exitCode=0
Sep 16 13:38:22.449582 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.449519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7" event={"ID":"2c77ae70-53da-4276-b67e-588b00283b0a","Type":"ContainerDied","Data":"9d02c3a2b3339a743feab3d591c336fe1d05df319730b9c9fe2a39c0cdea5a29"}
Sep 16 13:38:22.499471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.498878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:22.598359 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.598327    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:22.712158 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.712128    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:22.732010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.731987    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"]
Sep 16 13:38:22.733836 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:38:22.733812    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe934fe_558a_4c8a_82c0_339af02098f8.slice/crio-655aac269a4ed788adb91807fa6e6eed409cceee52b53d7f3f443637a66e14fd WatchSource:0}: Error finding container 655aac269a4ed788adb91807fa6e6eed409cceee52b53d7f3f443637a66e14fd: Status 404 returned error can't find the container with id 655aac269a4ed788adb91807fa6e6eed409cceee52b53d7f3f443637a66e14fd
Sep 16 13:38:22.800447 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:22.800427    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:23.458761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:23.458726    2556 generic.go:356] "Generic (PLEG): container finished" podID="2c77ae70-53da-4276-b67e-588b00283b0a" containerID="5cfba959d201651863b0e031777dbb02c0e2bbc2747966ae81526a25a2d91084" exitCode=0
Sep 16 13:38:23.459229 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:23.458844    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7" event={"ID":"2c77ae70-53da-4276-b67e-588b00283b0a","Type":"ContainerDied","Data":"5cfba959d201651863b0e031777dbb02c0e2bbc2747966ae81526a25a2d91084"}
Sep 16 13:38:23.460771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:23.460740    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72" event={"ID":"ebe934fe-558a-4c8a-82c0-339af02098f8","Type":"ContainerStarted","Data":"655aac269a4ed788adb91807fa6e6eed409cceee52b53d7f3f443637a66e14fd"}
Sep 16 13:38:24.877460 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:24.877430    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:25.004132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.004069    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-bundle\") pod \"2c77ae70-53da-4276-b67e-588b00283b0a\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") "
Sep 16 13:38:25.004326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.004200    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnnnt\" (UniqueName: \"kubernetes.io/projected/2c77ae70-53da-4276-b67e-588b00283b0a-kube-api-access-fnnnt\") pod \"2c77ae70-53da-4276-b67e-588b00283b0a\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") "
Sep 16 13:38:25.004326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.004285    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-util\") pod \"2c77ae70-53da-4276-b67e-588b00283b0a\" (UID: \"2c77ae70-53da-4276-b67e-588b00283b0a\") "
Sep 16 13:38:25.004935 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.004891    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-bundle" (OuterVolumeSpecName: "bundle") pod "2c77ae70-53da-4276-b67e-588b00283b0a" (UID: "2c77ae70-53da-4276-b67e-588b00283b0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:25.006829 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.006792    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c77ae70-53da-4276-b67e-588b00283b0a-kube-api-access-fnnnt" (OuterVolumeSpecName: "kube-api-access-fnnnt") pod "2c77ae70-53da-4276-b67e-588b00283b0a" (UID: "2c77ae70-53da-4276-b67e-588b00283b0a"). InnerVolumeSpecName "kube-api-access-fnnnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:38:25.013418 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.013369    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-util" (OuterVolumeSpecName: "util") pod "2c77ae70-53da-4276-b67e-588b00283b0a" (UID: "2c77ae70-53da-4276-b67e-588b00283b0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:38:25.105717 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.105676    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnnt\" (UniqueName: \"kubernetes.io/projected/2c77ae70-53da-4276-b67e-588b00283b0a-kube-api-access-fnnnt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:38:25.105717 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.105714    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:38:25.105886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.105728    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c77ae70-53da-4276-b67e-588b00283b0a-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:38:25.473702 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.473670    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7"
Sep 16 13:38:25.473857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.473697    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0bl6d7" event={"ID":"2c77ae70-53da-4276-b67e-588b00283b0a","Type":"ContainerDied","Data":"a2a8bf209f73be90cb18a5480d733e6e8bcd8b6554e479cfef8e18af20eba668"}
Sep 16 13:38:25.473857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.473753    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2a8bf209f73be90cb18a5480d733e6e8bcd8b6554e479cfef8e18af20eba668"
Sep 16 13:38:25.675758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.675724    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:25.740710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:25.740634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:26.479954 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:26.479919    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72" event={"ID":"ebe934fe-558a-4c8a-82c0-339af02098f8","Type":"ContainerStarted","Data":"728a81fe31f9d0049a6cbdbcfced923345e50d6f2fd76b978d532428a34ba9a2"}
Sep 16 13:38:27.485602 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:27.485564    2556 generic.go:356] "Generic (PLEG): container finished" podID="ebe934fe-558a-4c8a-82c0-339af02098f8" containerID="728a81fe31f9d0049a6cbdbcfced923345e50d6f2fd76b978d532428a34ba9a2" exitCode=0
Sep 16 13:38:27.486000 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:27.485655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72" event={"ID":"ebe934fe-558a-4c8a-82c0-339af02098f8","Type":"ContainerDied","Data":"728a81fe31f9d0049a6cbdbcfced923345e50d6f2fd76b978d532428a34ba9a2"}
Sep 16 13:38:28.492156 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:28.492117    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72" event={"ID":"ebe934fe-558a-4c8a-82c0-339af02098f8","Type":"ContainerStarted","Data":"95a4c29658f9661e90e2d23fed555b558233901023240333226f5259fd0e87bc"}
Sep 16 13:38:28.492545 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:28.492220    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:28.512656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:28.512604    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72" podStartSLOduration=2.142130015 podStartE2EDuration="6.512586869s" totalImagesPullingTime="3.326247207s" totalInitContainerRuntime="1.044209647s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:28.510229169 +0000 UTC m=+318.976166093" watchObservedRunningTime="2026-09-16 13:38:28.512586869 +0000 UTC m=+318.978523778"
Sep 16 13:38:39.499510 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:39.499479    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6444576fdb-54p72"
Sep 16 13:38:41.145693 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.141170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:41.290954 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.290919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:41.465117 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.465044    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:41.477451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477424    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"]
Sep 16 13:38:41.477767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477756    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="pull"
Sep 16 13:38:41.477826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477769    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="pull"
Sep 16 13:38:41.477826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477784    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="util"
Sep 16 13:38:41.477826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477790    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="util"
Sep 16 13:38:41.477826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477797    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="extract"
Sep 16 13:38:41.477826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477802    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="extract"
Sep 16 13:38:41.478059 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.477854    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c77ae70-53da-4276-b67e-588b00283b0a" containerName="extract"
Sep 16 13:38:41.483650 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.483633    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.486928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.486910    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 13:38:41.487962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.487946    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 13:38:41.493185 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.493165    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"]
Sep 16 13:38:41.651413 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.651374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e2d3bb-6b44-4da3-b1a9-f515693da377-tls-certs\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.651413 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.651410    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zc8v\" (UniqueName: \"kubernetes.io/projected/d8e2d3bb-6b44-4da3-b1a9-f515693da377-kube-api-access-2zc8v\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.651607 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.651434    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8e2d3bb-6b44-4da3-b1a9-f515693da377-tmp\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.736763 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.736686    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:41.751817 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.751794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e2d3bb-6b44-4da3-b1a9-f515693da377-tls-certs\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.751817 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.751824    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zc8v\" (UniqueName: \"kubernetes.io/projected/d8e2d3bb-6b44-4da3-b1a9-f515693da377-kube-api-access-2zc8v\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.751981 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.751852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8e2d3bb-6b44-4da3-b1a9-f515693da377-tmp\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.754324 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.754300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8e2d3bb-6b44-4da3-b1a9-f515693da377-tmp\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.754463 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.754447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8e2d3bb-6b44-4da3-b1a9-f515693da377-tls-certs\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.762245 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.762227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zc8v\" (UniqueName: \"kubernetes.io/projected/d8e2d3bb-6b44-4da3-b1a9-f515693da377-kube-api-access-2zc8v\") pod \"kube-auth-proxy-68bbd697f4-67sml\" (UID: \"d8e2d3bb-6b44-4da3-b1a9-f515693da377\") " pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:41.795207 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:41.795183    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"
Sep 16 13:38:42.064390 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.064369    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68bbd697f4-67sml"]
Sep 16 13:38:42.066366 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:38:42.066341    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e2d3bb_6b44_4da3_b1a9_f515693da377.slice/crio-9c3c76420743e7d2ac6cfab3eb128f44e2b38b7cf914cf8e4aab264b9aa9495c WatchSource:0}: Error finding container 9c3c76420743e7d2ac6cfab3eb128f44e2b38b7cf914cf8e4aab264b9aa9495c: Status 404 returned error can't find the container with id 9c3c76420743e7d2ac6cfab3eb128f44e2b38b7cf914cf8e4aab264b9aa9495c
Sep 16 13:38:42.557152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.557117    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml" event={"ID":"d8e2d3bb-6b44-4da3-b1a9-f515693da377","Type":"ContainerStarted","Data":"9c3c76420743e7d2ac6cfab3eb128f44e2b38b7cf914cf8e4aab264b9aa9495c"}
Sep 16 13:38:42.968742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.968676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:42.979605 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.979571    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"]
Sep 16 13:38:42.983566 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.983539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:42.985393 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.985368    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 13:38:42.985522 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.985455    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:38:42.985705 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.985680    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 13:38:42.985705 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.985692    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 13:38:42.985842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.985818    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6ml29\""
Sep 16 13:38:42.998107 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:42.998061    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"]
Sep 16 13:38:43.041145 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.041072    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:43.063528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063433    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.063528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.063766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.063766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063594    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8ktp\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-kube-api-access-z8ktp\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.063766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.063766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.063766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.063747    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164358 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164323    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164557 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164557 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164438    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164557 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164471    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164727 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164606    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ktp\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-kube-api-access-z8ktp\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164727 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164651    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.164727 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.164717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.165171 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.165125    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.166957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.166929    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.166957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.166943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.167425 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.167410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.167476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.167411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.172471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.172446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.173007 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.172984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ktp\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-kube-api-access-z8ktp\") pod \"istiod-openshift-gateway-7b7995d5cd-ztvvb\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.299164 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.299014    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:43.589019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.588988    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"]
Sep 16 13:38:43.934900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:43.934864    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:44.892980 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:38:44.892949    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8407e5_921a_4ec7_ba96_9b8bdaf46616.slice/crio-83bd86946493ae795a8e409763adadfe92c6e581360786f551a7b22260fb7862 WatchSource:0}: Error finding container 83bd86946493ae795a8e409763adadfe92c6e581360786f551a7b22260fb7862: Status 404 returned error can't find the container with id 83bd86946493ae795a8e409763adadfe92c6e581360786f551a7b22260fb7862
Sep 16 13:38:45.575555 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:45.575503    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml" event={"ID":"d8e2d3bb-6b44-4da3-b1a9-f515693da377","Type":"ContainerStarted","Data":"e116371c3ffc932227d6978a139110a360fc578879a3f4bf3e4ea884f80e7457"}
Sep 16 13:38:45.576805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:45.576781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" event={"ID":"ef8407e5-921a-4ec7-ba96-9b8bdaf46616","Type":"ContainerStarted","Data":"83bd86946493ae795a8e409763adadfe92c6e581360786f551a7b22260fb7862"}
Sep 16 13:38:45.591166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:45.591123    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68bbd697f4-67sml" podStartSLOduration=1.7122945 podStartE2EDuration="4.591108821s" totalImagesPullingTime="2.878814321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:45.589047592 +0000 UTC m=+336.054984500" watchObservedRunningTime="2026-09-16 13:38:45.591108821 +0000 UTC m=+336.057045720"
Sep 16 13:38:47.463656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:47.463615    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:38:47.463940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:47.463685    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:38:47.589938 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:47.589904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" event={"ID":"ef8407e5-921a-4ec7-ba96-9b8bdaf46616","Type":"ContainerStarted","Data":"11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d"}
Sep 16 13:38:47.590160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:47.589962    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:47.607657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:47.607608    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" podStartSLOduration=3.039138609 podStartE2EDuration="5.60759239s" totalImagesPullingTime="2.568453781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:38:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:38:47.606562 +0000 UTC m=+338.072498930" watchObservedRunningTime="2026-09-16 13:38:47.60759239 +0000 UTC m=+338.073529297"
Sep 16 13:38:48.499312 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:48.499269    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:48.596785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:48.596755    2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-ztvvb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:38:48.596923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:48.596810    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" podUID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:38:48.599849 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:48.599828    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:49.107361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:49.107326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:49.206625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:49.206593    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:49.305853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:49.305808    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:51.595913 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:51.595881    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:38:52.180030 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:52.180000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:52.280736 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:52.280698    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:52.381118 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:52.381053    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:52.481406 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:52.481323    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:52.775462 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:52.775384    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:38:53.515551 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:38:53.515515    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:02.336783 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.336253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:02.396109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.396064    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"]
Sep 16 13:39:02.399766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.399746    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:02.401786 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.401759    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 13:39:02.401913 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.401820    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dvtbl\""
Sep 16 13:39:02.406374 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.406351    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"]
Sep 16 13:39:02.411038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.410999    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"]
Sep 16 13:39:02.411177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.411136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:02.413166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.413147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 13:39:02.413306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.413276    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bsbrg\""
Sep 16 13:39:02.422006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.421948    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"]
Sep 16 13:39:02.425753 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.425727    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:02.429429 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.429396    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:02.429549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.429444    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c15b622-2c5b-4be7-8588-22fc547cc1ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:02.429549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.429495    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df54w\" (UniqueName: \"kubernetes.io/projected/932290b0-d3d3-4367-a0c0-4a980bd5b55e-kube-api-access-df54w\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:02.429549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.429530    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwqzf\" (UniqueName: \"kubernetes.io/projected/0c15b622-2c5b-4be7-8588-22fc547cc1ad-kube-api-access-lwqzf\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:02.530577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.530535    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:02.530788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.530588    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c15b622-2c5b-4be7-8588-22fc547cc1ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:02.530788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.530632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df54w\" (UniqueName: \"kubernetes.io/projected/932290b0-d3d3-4367-a0c0-4a980bd5b55e-kube-api-access-df54w\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:02.530788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.530657    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwqzf\" (UniqueName: \"kubernetes.io/projected/0c15b622-2c5b-4be7-8588-22fc547cc1ad-kube-api-access-lwqzf\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:02.530788 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:02.530703    2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 13:39:02.530788 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:02.530758    2556 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 13:39:02.531045 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:02.530831    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c15b622-2c5b-4be7-8588-22fc547cc1ad-cert podName:0c15b622-2c5b-4be7-8588-22fc547cc1ad nodeName:}" failed. No retries permitted until 2026-09-16 13:39:03.030809742 +0000 UTC m=+353.496746650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0c15b622-2c5b-4be7-8588-22fc547cc1ad-cert") pod "kserve-controller-manager-d6f4b65b8-kl2sd" (UID: "0c15b622-2c5b-4be7-8588-22fc547cc1ad") : secret "kserve-webhook-server-cert" not found
Sep 16 13:39:02.531045 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:02.530850    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert podName:932290b0-d3d3-4367-a0c0-4a980bd5b55e nodeName:}" failed. No retries permitted until 2026-09-16 13:39:03.030842107 +0000 UTC m=+353.496778993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert") pod "llmisvc-controller-manager-777c6bd7d5-vgmpj" (UID: "932290b0-d3d3-4367-a0c0-4a980bd5b55e") : secret "llmisvc-webhook-server-cert" not found
Sep 16 13:39:02.559740 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.559705    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwqzf\" (UniqueName: \"kubernetes.io/projected/0c15b622-2c5b-4be7-8588-22fc547cc1ad-kube-api-access-lwqzf\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:02.559849 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.559740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df54w\" (UniqueName: \"kubernetes.io/projected/932290b0-d3d3-4367-a0c0-4a980bd5b55e-kube-api-access-df54w\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:02.792245 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:02.792207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:03.035700 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.035662    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:03.035916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.035719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c15b622-2c5b-4be7-8588-22fc547cc1ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:03.035916 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:03.035810    2556 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 13:39:03.035916 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:03.035882    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert podName:932290b0-d3d3-4367-a0c0-4a980bd5b55e nodeName:}" failed. No retries permitted until 2026-09-16 13:39:04.035866679 +0000 UTC m=+354.501803566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert") pod "llmisvc-controller-manager-777c6bd7d5-vgmpj" (UID: "932290b0-d3d3-4367-a0c0-4a980bd5b55e") : secret "llmisvc-webhook-server-cert" not found
Sep 16 13:39:03.038491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.038455    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0c15b622-2c5b-4be7-8588-22fc547cc1ad-cert\") pod \"kserve-controller-manager-d6f4b65b8-kl2sd\" (UID: \"0c15b622-2c5b-4be7-8588-22fc547cc1ad\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:03.314917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.314880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:03.615684 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.615658    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"]
Sep 16 13:39:03.618289 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:39:03.618260    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c15b622_2c5b_4be7_8588_22fc547cc1ad.slice/crio-dd579e4015181f6d10af549209536707e73c55424e23fc24714f5c522cd025bf WatchSource:0}: Error finding container dd579e4015181f6d10af549209536707e73c55424e23fc24714f5c522cd025bf: Status 404 returned error can't find the container with id dd579e4015181f6d10af549209536707e73c55424e23fc24714f5c522cd025bf
Sep 16 13:39:03.661742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.661707    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:03.666359 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.666329    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd" event={"ID":"0c15b622-2c5b-4be7-8588-22fc547cc1ad","Type":"ContainerStarted","Data":"dd579e4015181f6d10af549209536707e73c55424e23fc24714f5c522cd025bf"}
Sep 16 13:39:03.752150 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.752119    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:03.807505 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.807454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:03.980000 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.979962    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"]
Sep 16 13:39:03.980192 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:03.980168    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" podUID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" containerName="discovery" containerID="cri-o://11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d" gracePeriod=30
Sep 16 13:39:04.044257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.044233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:04.046708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.046680    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/932290b0-d3d3-4367-a0c0-4a980bd5b55e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vgmpj\" (UID: \"932290b0-d3d3-4367-a0c0-4a980bd5b55e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:04.224180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.224157    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:39:04.228344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.228318    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247650    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-kubeconfig\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-cacerts\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247739    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-local-certs\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247813    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-token\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247840    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-dns-cert\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247883    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8ktp\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-kube-api-access-z8ktp\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.247911    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-ca-configmap\") pod \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\" (UID: \"ef8407e5-921a-4ec7-ba96-9b8bdaf46616\") "
Sep 16 13:39:04.248788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.248519    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:39:04.261483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.261435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:39:04.261762 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.261732    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-cacerts" (OuterVolumeSpecName: "cacerts") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:39:04.261959 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.261765    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:39:04.261959 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.261783    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-kube-api-access-z8ktp" (OuterVolumeSpecName: "kube-api-access-z8ktp") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "kube-api-access-z8ktp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:39:04.262118 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.262030    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:39:04.263363 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.263319    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-local-certs" (OuterVolumeSpecName: "local-certs") pod "ef8407e5-921a-4ec7-ba96-9b8bdaf46616" (UID: "ef8407e5-921a-4ec7-ba96-9b8bdaf46616"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:39:04.349066 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349035    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8ktp\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-kube-api-access-z8ktp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.349066 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349069    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-ca-configmap\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.349286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349084    2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-kubeconfig\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.349286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349112    2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-cacerts\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.349286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349125    2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-local-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.349286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349139    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.349286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.349147    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ef8407e5-921a-4ec7-ba96-9b8bdaf46616-istio-csr-dns-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:39:04.527575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.527544    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"]
Sep 16 13:39:04.528792 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:39:04.528766    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod932290b0_d3d3_4367_a0c0_4a980bd5b55e.slice/crio-61d2a1c05b5aaabee3c5244cdfd2641c099ae30690add582aad23eb804dae22f WatchSource:0}: Error finding container 61d2a1c05b5aaabee3c5244cdfd2641c099ae30690add582aad23eb804dae22f: Status 404 returned error can't find the container with id 61d2a1c05b5aaabee3c5244cdfd2641c099ae30690add582aad23eb804dae22f
Sep 16 13:39:04.674210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.674168    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj" event={"ID":"932290b0-d3d3-4367-a0c0-4a980bd5b55e","Type":"ContainerStarted","Data":"61d2a1c05b5aaabee3c5244cdfd2641c099ae30690add582aad23eb804dae22f"}
Sep 16 13:39:04.675802 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.675773    2556 generic.go:356] "Generic (PLEG): container finished" podID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" containerID="11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d" exitCode=0
Sep 16 13:39:04.675940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.675806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" event={"ID":"ef8407e5-921a-4ec7-ba96-9b8bdaf46616","Type":"ContainerDied","Data":"11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d"}
Sep 16 13:39:04.675940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.675822    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"
Sep 16 13:39:04.675940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.675863    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb" event={"ID":"ef8407e5-921a-4ec7-ba96-9b8bdaf46616","Type":"ContainerDied","Data":"83bd86946493ae795a8e409763adadfe92c6e581360786f551a7b22260fb7862"}
Sep 16 13:39:04.675940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.675894    2556 scope.go:117] "RemoveContainer" containerID="11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d"
Sep 16 13:39:04.686597 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.686580    2556 scope.go:117] "RemoveContainer" containerID="11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d"
Sep 16 13:39:04.686865 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:39:04.686847    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d\": container with ID starting with 11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d not found: ID does not exist" containerID="11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d"
Sep 16 13:39:04.686919 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.686874    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d"} err="failed to get container status \"11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d\": rpc error: code = NotFound desc = could not find container \"11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d\": container with ID starting with 11c5f77c996e54a22db895e34d402019b7dae54819dfdb12f72f1fb5ce4d093d not found: ID does not exist"
Sep 16 13:39:04.699881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.699856    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"]
Sep 16 13:39:04.704519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:04.704493    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ztvvb"]
Sep 16 13:39:06.060237 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:06.060209    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" path="/var/lib/kubelet/pods/ef8407e5-921a-4ec7-ba96-9b8bdaf46616/volumes"
Sep 16 13:39:06.689688 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:06.689655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd" event={"ID":"0c15b622-2c5b-4be7-8588-22fc547cc1ad","Type":"ContainerStarted","Data":"1c026a2731b6aa514a6cf14dc04455b3464dd7862ce5cab7959b698c0a5d1972"}
Sep 16 13:39:06.689869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:06.689730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:06.706934 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:06.706879    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd" podStartSLOduration=2.428547629 podStartE2EDuration="4.706861055s" totalImagesPullingTime="2.278313426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:06.706520537 +0000 UTC m=+357.172457505" watchObservedRunningTime="2026-09-16 13:39:06.706861055 +0000 UTC m=+357.172797963"
Sep 16 13:39:07.153368 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:07.153288    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:08.705262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:08.705224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj" event={"ID":"932290b0-d3d3-4367-a0c0-4a980bd5b55e","Type":"ContainerStarted","Data":"220b4a297768d4bedec875487752f194c085e0a2e37338cf0aaba12bf0cd67ed"}
Sep 16 13:39:08.705715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:08.705342    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:08.721076 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:08.721028    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj" podStartSLOduration=3.377161378 podStartE2EDuration="6.721011284s" totalImagesPullingTime="3.343849906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:39:08.72037148 +0000 UTC m=+359.186308400" watchObservedRunningTime="2026-09-16 13:39:08.721011284 +0000 UTC m=+359.186948191"
Sep 16 13:39:37.704675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:37.704640    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kl2sd"
Sep 16 13:39:39.713108 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:39.713061    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vgmpj"
Sep 16 13:39:53.259599 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:53.259561    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:53.364724 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:53.364679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:39:56.726525 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.726488    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"]
Sep 16 13:39:56.726902 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.726857    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" containerName="discovery"
Sep 16 13:39:56.726902 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.726869    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" containerName="discovery"
Sep 16 13:39:56.726972 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.726937    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8407e5-921a-4ec7-ba96-9b8bdaf46616" containerName="discovery"
Sep 16 13:39:56.733955 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.733935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.736265 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.736243    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 13:39:56.736780 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.736759    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 13:39:56.736905 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.736829    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2mrld\""
Sep 16 13:39:56.737553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.737533    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"]
Sep 16 13:39:56.828724 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.828693    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"]
Sep 16 13:39:56.832444 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.832426    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:56.839238 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.839216    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"]
Sep 16 13:39:56.886994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.886962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.887171 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.886996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-665mt\" (UniqueName: \"kubernetes.io/projected/34261a00-5c31-4a57-a22f-3417ba9b25db-kube-api-access-665mt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.887250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.887188    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.929997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.929959    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"]
Sep 16 13:39:56.935835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.935811    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:56.939876 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.939854    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"]
Sep 16 13:39:56.987896 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.987831    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:56.987896 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.987871    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.987896 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.987896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.988074 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.987917    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-665mt\" (UniqueName: \"kubernetes.io/projected/34261a00-5c31-4a57-a22f-3417ba9b25db-kube-api-access-665mt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.988074 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.988003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:56.988184 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.988076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsn46\" (UniqueName: \"kubernetes.io/projected/80d53f53-3355-471b-94d2-4b899e2fa9ae-kube-api-access-rsn46\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:56.988298 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.988281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.988345 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.988331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:56.995361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:56.995336    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-665mt\" (UniqueName: \"kubernetes.io/projected/34261a00-5c31-4a57-a22f-3417ba9b25db-kube-api-access-665mt\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:57.031915 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.031881    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"]
Sep 16 13:39:57.035686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.035670    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.043113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.043062    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"]
Sep 16 13:39:57.045183 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.045155    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:39:57.089151 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089110    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.089304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf46h\" (UniqueName: \"kubernetes.io/projected/420b79f3-cc10-47c7-849a-6da9d8085336-kube-api-access-kf46h\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.089304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsn46\" (UniqueName: \"kubernetes.io/projected/80d53f53-3355-471b-94d2-4b899e2fa9ae-kube-api-access-rsn46\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.089410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.089410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.089517 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089423    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.089569 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.089739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.089714    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.098190 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.098139    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsn46\" (UniqueName: \"kubernetes.io/projected/80d53f53-3355-471b-94d2-4b899e2fa9ae-kube-api-access-rsn46\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.141619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.141581    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:39:57.191203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.190374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf46h\" (UniqueName: \"kubernetes.io/projected/420b79f3-cc10-47c7-849a-6da9d8085336-kube-api-access-kf46h\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.191203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.190432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.191203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.190456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.191203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.190657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbncv\" (UniqueName: \"kubernetes.io/projected/01623565-2a88-4212-867a-82d5874f2131-kube-api-access-dbncv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.191203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.190813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.191203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.190853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.191728 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.191479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.191797 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.191732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.199929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.199454    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf46h\" (UniqueName: \"kubernetes.io/projected/420b79f3-cc10-47c7-849a-6da9d8085336-kube-api-access-kf46h\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.248419 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.248335    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:39:57.291314 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.291278    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.291314 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.291313    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.291568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.291342    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbncv\" (UniqueName: \"kubernetes.io/projected/01623565-2a88-4212-867a-82d5874f2131-kube-api-access-dbncv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.291728 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.291707    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.291791 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.291730    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.300721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.300679    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbncv\" (UniqueName: \"kubernetes.io/projected/01623565-2a88-4212-867a-82d5874f2131-kube-api-access-dbncv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.321638 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.321598    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"]
Sep 16 13:39:57.322908 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:39:57.322879    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34261a00_5c31_4a57_a22f_3417ba9b25db.slice/crio-cb9502e0bdb857b540b3729e61a5635e0058431e5dfb24714e1b04a1d2cecadb WatchSource:0}: Error finding container cb9502e0bdb857b540b3729e61a5635e0058431e5dfb24714e1b04a1d2cecadb: Status 404 returned error can't find the container with id cb9502e0bdb857b540b3729e61a5635e0058431e5dfb24714e1b04a1d2cecadb
Sep 16 13:39:57.346201 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.346171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:39:57.416872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.416837    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"]
Sep 16 13:39:57.420283 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:39:57.420242    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d53f53_3355_471b_94d2_4b899e2fa9ae.slice/crio-828eae45c4570cdcc0b9038de063c06497c51f64a12beb6d3e9f6cf79ee63a76 WatchSource:0}: Error finding container 828eae45c4570cdcc0b9038de063c06497c51f64a12beb6d3e9f6cf79ee63a76: Status 404 returned error can't find the container with id 828eae45c4570cdcc0b9038de063c06497c51f64a12beb6d3e9f6cf79ee63a76
Sep 16 13:39:57.519544 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.519520    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"]
Sep 16 13:39:57.521669 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:39:57.521640    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420b79f3_cc10_47c7_849a_6da9d8085336.slice/crio-1ec269dcad476f82e5f2f67f503eea41e7fd5cd0794e5ec5c84c262f366dc7ec WatchSource:0}: Error finding container 1ec269dcad476f82e5f2f67f503eea41e7fd5cd0794e5ec5c84c262f366dc7ec: Status 404 returned error can't find the container with id 1ec269dcad476f82e5f2f67f503eea41e7fd5cd0794e5ec5c84c262f366dc7ec
Sep 16 13:39:57.622565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.622536    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"]
Sep 16 13:39:57.623968 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:39:57.623940    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01623565_2a88_4212_867a_82d5874f2131.slice/crio-c7b29b08ad1e257d9a6375f3fa4041f5b9982f48b64435e0fc4f13aa1ea7be22 WatchSource:0}: Error finding container c7b29b08ad1e257d9a6375f3fa4041f5b9982f48b64435e0fc4f13aa1ea7be22: Status 404 returned error can't find the container with id c7b29b08ad1e257d9a6375f3fa4041f5b9982f48b64435e0fc4f13aa1ea7be22
Sep 16 13:39:57.948187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.948150    2556 generic.go:356] "Generic (PLEG): container finished" podID="420b79f3-cc10-47c7-849a-6da9d8085336" containerID="f7b0c9ab9e8b8e00d67828f0b85970117b25fd28ae201e78751375d1b9a277cd" exitCode=0
Sep 16 13:39:57.948633 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.948235    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b" event={"ID":"420b79f3-cc10-47c7-849a-6da9d8085336","Type":"ContainerDied","Data":"f7b0c9ab9e8b8e00d67828f0b85970117b25fd28ae201e78751375d1b9a277cd"}
Sep 16 13:39:57.948633 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.948287    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b" event={"ID":"420b79f3-cc10-47c7-849a-6da9d8085336","Type":"ContainerStarted","Data":"1ec269dcad476f82e5f2f67f503eea41e7fd5cd0794e5ec5c84c262f366dc7ec"}
Sep 16 13:39:57.950003 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.949978    2556 generic.go:356] "Generic (PLEG): container finished" podID="01623565-2a88-4212-867a-82d5874f2131" containerID="f2314f4d1e3b8281038b2001b14dc64809ceb91b58bf7addeebf09f86e9e2e02" exitCode=0
Sep 16 13:39:57.950118 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.950062    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk" event={"ID":"01623565-2a88-4212-867a-82d5874f2131","Type":"ContainerDied","Data":"f2314f4d1e3b8281038b2001b14dc64809ceb91b58bf7addeebf09f86e9e2e02"}
Sep 16 13:39:57.950185 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.950120    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk" event={"ID":"01623565-2a88-4212-867a-82d5874f2131","Type":"ContainerStarted","Data":"c7b29b08ad1e257d9a6375f3fa4041f5b9982f48b64435e0fc4f13aa1ea7be22"}
Sep 16 13:39:57.951680 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.951654    2556 generic.go:356] "Generic (PLEG): container finished" podID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerID="7a2a44003a0fa1915fcc63cbe86074ea2417f5f3e2add33b462f351ee32f7d39" exitCode=0
Sep 16 13:39:57.951762 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.951741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb" event={"ID":"34261a00-5c31-4a57-a22f-3417ba9b25db","Type":"ContainerDied","Data":"7a2a44003a0fa1915fcc63cbe86074ea2417f5f3e2add33b462f351ee32f7d39"}
Sep 16 13:39:57.951808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.951785    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb" event={"ID":"34261a00-5c31-4a57-a22f-3417ba9b25db","Type":"ContainerStarted","Data":"cb9502e0bdb857b540b3729e61a5635e0058431e5dfb24714e1b04a1d2cecadb"}
Sep 16 13:39:57.953512 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.953482    2556 generic.go:356] "Generic (PLEG): container finished" podID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerID="72a68a8017ae61a56599ddcad3799ef39dc54b645f990cce1fcb7f7d473b6e49" exitCode=0
Sep 16 13:39:57.953590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.953531    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h" event={"ID":"80d53f53-3355-471b-94d2-4b899e2fa9ae","Type":"ContainerDied","Data":"72a68a8017ae61a56599ddcad3799ef39dc54b645f990cce1fcb7f7d473b6e49"}
Sep 16 13:39:57.953590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:57.953580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h" event={"ID":"80d53f53-3355-471b-94d2-4b899e2fa9ae","Type":"ContainerStarted","Data":"828eae45c4570cdcc0b9038de063c06497c51f64a12beb6d3e9f6cf79ee63a76"}
Sep 16 13:39:58.963011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:58.962935    2556 generic.go:356] "Generic (PLEG): container finished" podID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerID="ce12a6944f8aa0e480ff6c635e4b8833290a2a27316ba011f6509787b6b3c1cb" exitCode=0
Sep 16 13:39:58.963011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:58.963001    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h" event={"ID":"80d53f53-3355-471b-94d2-4b899e2fa9ae","Type":"ContainerDied","Data":"ce12a6944f8aa0e480ff6c635e4b8833290a2a27316ba011f6509787b6b3c1cb"}
Sep 16 13:39:59.969811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:59.969773    2556 generic.go:356] "Generic (PLEG): container finished" podID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerID="531c80f1b438794140c1549c14b9c916d8b8251b42937a307bccb460f4ec7428" exitCode=0
Sep 16 13:39:59.970266 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:59.969860    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h" event={"ID":"80d53f53-3355-471b-94d2-4b899e2fa9ae","Type":"ContainerDied","Data":"531c80f1b438794140c1549c14b9c916d8b8251b42937a307bccb460f4ec7428"}
Sep 16 13:39:59.971725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:59.971704    2556 generic.go:356] "Generic (PLEG): container finished" podID="420b79f3-cc10-47c7-849a-6da9d8085336" containerID="4cc6fb2e8d68cf47bc4533c899102f8689917042a195d68bcf6645e2abb59c51" exitCode=0
Sep 16 13:39:59.971843 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:59.971789    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b" event={"ID":"420b79f3-cc10-47c7-849a-6da9d8085336","Type":"ContainerDied","Data":"4cc6fb2e8d68cf47bc4533c899102f8689917042a195d68bcf6645e2abb59c51"}
Sep 16 13:39:59.973699 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:59.973674    2556 generic.go:356] "Generic (PLEG): container finished" podID="01623565-2a88-4212-867a-82d5874f2131" containerID="a1527d713e949ae9e1d42b8255dc7803475200cd251bd003a78cae2634d9dd1f" exitCode=0
Sep 16 13:39:59.973827 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:39:59.973720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk" event={"ID":"01623565-2a88-4212-867a-82d5874f2131","Type":"ContainerDied","Data":"a1527d713e949ae9e1d42b8255dc7803475200cd251bd003a78cae2634d9dd1f"}
Sep 16 13:40:00.981137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:00.981098    2556 generic.go:356] "Generic (PLEG): container finished" podID="420b79f3-cc10-47c7-849a-6da9d8085336" containerID="0d44019d01e710dbc45324b4a909cfce63a19ef01f45c6e62fe99ae74601d6f7" exitCode=0
Sep 16 13:40:00.981577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:00.981185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b" event={"ID":"420b79f3-cc10-47c7-849a-6da9d8085336","Type":"ContainerDied","Data":"0d44019d01e710dbc45324b4a909cfce63a19ef01f45c6e62fe99ae74601d6f7"}
Sep 16 13:40:00.983257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:00.983237    2556 generic.go:356] "Generic (PLEG): container finished" podID="01623565-2a88-4212-867a-82d5874f2131" containerID="68e8f6b604f38508fe56c9d97aef83a01b9cb2f88fa6f125c4b72d841bdcd189" exitCode=0
Sep 16 13:40:00.983354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:00.983330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk" event={"ID":"01623565-2a88-4212-867a-82d5874f2131","Type":"ContainerDied","Data":"68e8f6b604f38508fe56c9d97aef83a01b9cb2f88fa6f125c4b72d841bdcd189"}
Sep 16 13:40:01.116726 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.116703    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:40:01.224549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.224515    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-bundle\") pod \"80d53f53-3355-471b-94d2-4b899e2fa9ae\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") "
Sep 16 13:40:01.224693 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.224603    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-util\") pod \"80d53f53-3355-471b-94d2-4b899e2fa9ae\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") "
Sep 16 13:40:01.224693 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.224660    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsn46\" (UniqueName: \"kubernetes.io/projected/80d53f53-3355-471b-94d2-4b899e2fa9ae-kube-api-access-rsn46\") pod \"80d53f53-3355-471b-94d2-4b899e2fa9ae\" (UID: \"80d53f53-3355-471b-94d2-4b899e2fa9ae\") "
Sep 16 13:40:01.225023 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.224986    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-bundle" (OuterVolumeSpecName: "bundle") pod "80d53f53-3355-471b-94d2-4b899e2fa9ae" (UID: "80d53f53-3355-471b-94d2-4b899e2fa9ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:01.226861 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.226841    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d53f53-3355-471b-94d2-4b899e2fa9ae-kube-api-access-rsn46" (OuterVolumeSpecName: "kube-api-access-rsn46") pod "80d53f53-3355-471b-94d2-4b899e2fa9ae" (UID: "80d53f53-3355-471b-94d2-4b899e2fa9ae"). InnerVolumeSpecName "kube-api-access-rsn46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:01.232672 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.232623    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-util" (OuterVolumeSpecName: "util") pod "80d53f53-3355-471b-94d2-4b899e2fa9ae" (UID: "80d53f53-3355-471b-94d2-4b899e2fa9ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:01.326219 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.326185    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:01.326219 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.326211    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsn46\" (UniqueName: \"kubernetes.io/projected/80d53f53-3355-471b-94d2-4b899e2fa9ae-kube-api-access-rsn46\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:01.326219 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.326220    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80d53f53-3355-471b-94d2-4b899e2fa9ae-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:01.636857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.636782    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5947c4777d-hpmgb"]
Sep 16 13:40:01.637154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637141    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="pull"
Sep 16 13:40:01.637206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637155    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="pull"
Sep 16 13:40:01.637206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637168    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="util"
Sep 16 13:40:01.637206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637173    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="util"
Sep 16 13:40:01.637206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637189    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="extract"
Sep 16 13:40:01.637206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637194    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="extract"
Sep 16 13:40:01.637375 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.637264    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d53f53-3355-471b-94d2-4b899e2fa9ae" containerName="extract"
Sep 16 13:40:01.640397 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.640381    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.654989 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.654962    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5947c4777d-hpmgb"]
Sep 16 13:40:01.687198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.687172    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:01.729503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729469    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-trusted-ca-bundle\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.729667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-service-ca\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.729667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-oauth-serving-cert\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.729667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729623    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9clk\" (UniqueName: \"kubernetes.io/projected/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-kube-api-access-f9clk\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.729667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-config\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.729667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-serving-cert\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.729828 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.729681    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-oauth-config\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.737613 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.737589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:01.830159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-service-ca\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830167    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-oauth-serving-cert\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830185    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9clk\" (UniqueName: \"kubernetes.io/projected/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-kube-api-access-f9clk\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-config\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-serving-cert\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830268    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-oauth-config\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830317    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-trusted-ca-bundle\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.830992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.830961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-service-ca\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.831254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.831002    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-oauth-serving-cert\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.831254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.831040    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-config\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.831254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.831189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-trusted-ca-bundle\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.832910 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.832890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-serving-cert\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.832910 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.832907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-console-oauth-config\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.838815 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.838797    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9clk\" (UniqueName: \"kubernetes.io/projected/31cb106b-fb16-4ac7-a5b9-2bc9659b97be-kube-api-access-f9clk\") pod \"console-5947c4777d-hpmgb\" (UID: \"31cb106b-fb16-4ac7-a5b9-2bc9659b97be\") " pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.950413 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.950376    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:01.992807 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.992763    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h" event={"ID":"80d53f53-3355-471b-94d2-4b899e2fa9ae","Type":"ContainerDied","Data":"828eae45c4570cdcc0b9038de063c06497c51f64a12beb6d3e9f6cf79ee63a76"}
Sep 16 13:40:01.993286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.992861    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828eae45c4570cdcc0b9038de063c06497c51f64a12beb6d3e9f6cf79ee63a76"
Sep 16 13:40:01.993286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:01.992954    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d749w4h"
Sep 16 13:40:02.160283 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.160258    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:40:02.189081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.189059    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:40:02.245132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.245107    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5947c4777d-hpmgb"]
Sep 16 13:40:02.246425 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:02.246392    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cb106b_fb16_4ac7_a5b9_2bc9659b97be.slice/crio-bf4d66d492dcdb4d6505ca18f88dd366bf058049c627396e8e82c7374875e1e2 WatchSource:0}: Error finding container bf4d66d492dcdb4d6505ca18f88dd366bf058049c627396e8e82c7374875e1e2: Status 404 returned error can't find the container with id bf4d66d492dcdb4d6505ca18f88dd366bf058049c627396e8e82c7374875e1e2
Sep 16 13:40:02.345454 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.345414    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-bundle\") pod \"01623565-2a88-4212-867a-82d5874f2131\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") "
Sep 16 13:40:02.345628 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.345526    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-util\") pod \"420b79f3-cc10-47c7-849a-6da9d8085336\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") "
Sep 16 13:40:02.345628 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.345563    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-bundle\") pod \"420b79f3-cc10-47c7-849a-6da9d8085336\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") "
Sep 16 13:40:02.345628 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.345608    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbncv\" (UniqueName: \"kubernetes.io/projected/01623565-2a88-4212-867a-82d5874f2131-kube-api-access-dbncv\") pod \"01623565-2a88-4212-867a-82d5874f2131\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") "
Sep 16 13:40:02.345791 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.345648    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-util\") pod \"01623565-2a88-4212-867a-82d5874f2131\" (UID: \"01623565-2a88-4212-867a-82d5874f2131\") "
Sep 16 13:40:02.345791 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.345698    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf46h\" (UniqueName: \"kubernetes.io/projected/420b79f3-cc10-47c7-849a-6da9d8085336-kube-api-access-kf46h\") pod \"420b79f3-cc10-47c7-849a-6da9d8085336\" (UID: \"420b79f3-cc10-47c7-849a-6da9d8085336\") "
Sep 16 13:40:02.346080 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.346051    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-bundle" (OuterVolumeSpecName: "bundle") pod "420b79f3-cc10-47c7-849a-6da9d8085336" (UID: "420b79f3-cc10-47c7-849a-6da9d8085336"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:02.346354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.346283    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-bundle" (OuterVolumeSpecName: "bundle") pod "01623565-2a88-4212-867a-82d5874f2131" (UID: "01623565-2a88-4212-867a-82d5874f2131"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:02.348296 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.348262    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01623565-2a88-4212-867a-82d5874f2131-kube-api-access-dbncv" (OuterVolumeSpecName: "kube-api-access-dbncv") pod "01623565-2a88-4212-867a-82d5874f2131" (UID: "01623565-2a88-4212-867a-82d5874f2131"). InnerVolumeSpecName "kube-api-access-dbncv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:02.348468 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.348446    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420b79f3-cc10-47c7-849a-6da9d8085336-kube-api-access-kf46h" (OuterVolumeSpecName: "kube-api-access-kf46h") pod "420b79f3-cc10-47c7-849a-6da9d8085336" (UID: "420b79f3-cc10-47c7-849a-6da9d8085336"). InnerVolumeSpecName "kube-api-access-kf46h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:02.353902 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.353875    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-util" (OuterVolumeSpecName: "util") pod "420b79f3-cc10-47c7-849a-6da9d8085336" (UID: "420b79f3-cc10-47c7-849a-6da9d8085336"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:02.355144 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.355119    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-util" (OuterVolumeSpecName: "util") pod "01623565-2a88-4212-867a-82d5874f2131" (UID: "01623565-2a88-4212-867a-82d5874f2131"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:02.446939 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.446904    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf46h\" (UniqueName: \"kubernetes.io/projected/420b79f3-cc10-47c7-849a-6da9d8085336-kube-api-access-kf46h\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:02.446939 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.446941    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:02.447125 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.446954    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:02.447125 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.446963    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/420b79f3-cc10-47c7-849a-6da9d8085336-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:02.447125 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.446970    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbncv\" (UniqueName: \"kubernetes.io/projected/01623565-2a88-4212-867a-82d5874f2131-kube-api-access-dbncv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:02.447125 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.446979    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01623565-2a88-4212-867a-82d5874f2131-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:02.999288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.999251    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b"
Sep 16 13:40:02.999679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.999253    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6kns6b" event={"ID":"420b79f3-cc10-47c7-849a-6da9d8085336","Type":"ContainerDied","Data":"1ec269dcad476f82e5f2f67f503eea41e7fd5cd0794e5ec5c84c262f366dc7ec"}
Sep 16 13:40:02.999679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:02.999368    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ec269dcad476f82e5f2f67f503eea41e7fd5cd0794e5ec5c84c262f366dc7ec"
Sep 16 13:40:03.000910 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:03.000883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5947c4777d-hpmgb" event={"ID":"31cb106b-fb16-4ac7-a5b9-2bc9659b97be","Type":"ContainerStarted","Data":"85c8f0758524b0fbfd9d338427a512c78402387bcf7d8e8ceac67e4c3e2579dd"}
Sep 16 13:40:03.001050 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:03.000929    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5947c4777d-hpmgb" event={"ID":"31cb106b-fb16-4ac7-a5b9-2bc9659b97be","Type":"ContainerStarted","Data":"bf4d66d492dcdb4d6505ca18f88dd366bf058049c627396e8e82c7374875e1e2"}
Sep 16 13:40:03.002842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:03.002823    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk"
Sep 16 13:40:03.002943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:03.002858    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afvnbk" event={"ID":"01623565-2a88-4212-867a-82d5874f2131","Type":"ContainerDied","Data":"c7b29b08ad1e257d9a6375f3fa4041f5b9982f48b64435e0fc4f13aa1ea7be22"}
Sep 16 13:40:03.002943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:03.002892    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7b29b08ad1e257d9a6375f3fa4041f5b9982f48b64435e0fc4f13aa1ea7be22"
Sep 16 13:40:03.017881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:03.017828    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5947c4777d-hpmgb" podStartSLOduration=2.01781387 podStartE2EDuration="2.01781387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:03.01633745 +0000 UTC m=+413.482274357" watchObservedRunningTime="2026-09-16 13:40:03.01781387 +0000 UTC m=+413.483750835"
Sep 16 13:40:04.011179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:04.011141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb" event={"ID":"34261a00-5c31-4a57-a22f-3417ba9b25db","Type":"ContainerStarted","Data":"c2f29163050699aab3e1f0eddafe09b48343c8d7cc38ad3b134e9986a4e630bd"}
Sep 16 13:40:05.018519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:05.018484    2556 generic.go:356] "Generic (PLEG): container finished" podID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerID="c2f29163050699aab3e1f0eddafe09b48343c8d7cc38ad3b134e9986a4e630bd" exitCode=0
Sep 16 13:40:05.018953 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:05.018569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb" event={"ID":"34261a00-5c31-4a57-a22f-3417ba9b25db","Type":"ContainerDied","Data":"c2f29163050699aab3e1f0eddafe09b48343c8d7cc38ad3b134e9986a4e630bd"}
Sep 16 13:40:06.026102 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:06.026047    2556 generic.go:356] "Generic (PLEG): container finished" podID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerID="96c74f1bc1781a9c0bde6c0d0a58a7358147d55fc4fce0e181dc201232157930" exitCode=0
Sep 16 13:40:06.026467 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:06.026144    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb" event={"ID":"34261a00-5c31-4a57-a22f-3417ba9b25db","Type":"ContainerDied","Data":"96c74f1bc1781a9c0bde6c0d0a58a7358147d55fc4fce0e181dc201232157930"}
Sep 16 13:40:07.165573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.165549    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:40:07.186480 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.186451    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-665mt\" (UniqueName: \"kubernetes.io/projected/34261a00-5c31-4a57-a22f-3417ba9b25db-kube-api-access-665mt\") pod \"34261a00-5c31-4a57-a22f-3417ba9b25db\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") "
Sep 16 13:40:07.186617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.186501    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-bundle\") pod \"34261a00-5c31-4a57-a22f-3417ba9b25db\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") "
Sep 16 13:40:07.186617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.186581    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-util\") pod \"34261a00-5c31-4a57-a22f-3417ba9b25db\" (UID: \"34261a00-5c31-4a57-a22f-3417ba9b25db\") "
Sep 16 13:40:07.187046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.187013    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-bundle" (OuterVolumeSpecName: "bundle") pod "34261a00-5c31-4a57-a22f-3417ba9b25db" (UID: "34261a00-5c31-4a57-a22f-3417ba9b25db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:07.188678 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.188648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34261a00-5c31-4a57-a22f-3417ba9b25db-kube-api-access-665mt" (OuterVolumeSpecName: "kube-api-access-665mt") pod "34261a00-5c31-4a57-a22f-3417ba9b25db" (UID: "34261a00-5c31-4a57-a22f-3417ba9b25db"). InnerVolumeSpecName "kube-api-access-665mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:07.194528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.194500    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-util" (OuterVolumeSpecName: "util") pod "34261a00-5c31-4a57-a22f-3417ba9b25db" (UID: "34261a00-5c31-4a57-a22f-3417ba9b25db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:07.287379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.287295    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:07.287379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.287325    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34261a00-5c31-4a57-a22f-3417ba9b25db-util\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:07.287379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:07.287334    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-665mt\" (UniqueName: \"kubernetes.io/projected/34261a00-5c31-4a57-a22f-3417ba9b25db-kube-api-access-665mt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:08.038352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:08.038328    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb"
Sep 16 13:40:08.038544 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:08.038325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cgncdb" event={"ID":"34261a00-5c31-4a57-a22f-3417ba9b25db","Type":"ContainerDied","Data":"cb9502e0bdb857b540b3729e61a5635e0058431e5dfb24714e1b04a1d2cecadb"}
Sep 16 13:40:08.038544 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:08.038451    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9502e0bdb857b540b3729e61a5635e0058431e5dfb24714e1b04a1d2cecadb"
Sep 16 13:40:11.068233 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:11.068193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:11.851434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:11.851396    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:11.951387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:11.951344    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:11.951588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:11.951454    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:11.958131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:11.958104    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:12.062885 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:12.062855    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5947c4777d-hpmgb"
Sep 16 13:40:12.145924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:12.145800    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57785d5cf8-m929f"]
Sep 16 13:40:12.259611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:12.259578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:12.471467 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:12.471436    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:12.563617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:12.563583    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:14.319566 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319536    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-886x4"]
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319870    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="util"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319881    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="util"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319890    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="extract"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319895    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="extract"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319901    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="pull"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319906    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="pull"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319917    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="extract"
Sep 16 13:40:14.319924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319924    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="extract"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319932    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="util"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319937    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="util"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319944    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="pull"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319951    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="pull"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319959    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="util"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319964    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="util"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319973    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="pull"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319979    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="pull"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319988    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="extract"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.319993    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="extract"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.320046    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="01623565-2a88-4212-867a-82d5874f2131" containerName="extract"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.320052    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="420b79f3-cc10-47c7-849a-6da9d8085336" containerName="extract"
Sep 16 13:40:14.320344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.320060    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="34261a00-5c31-4a57-a22f-3417ba9b25db" containerName="extract"
Sep 16 13:40:14.324249 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.324233    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:14.326268 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.326244    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 13:40:14.326374 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.326290    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5jffd\""
Sep 16 13:40:14.326374 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.326310    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 13:40:14.333835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.333817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflpb\" (UniqueName: \"kubernetes.io/projected/e5ca2022-9943-4720-b501-d802020e8b4e-kube-api-access-pflpb\") pod \"authorino-operator-76b478bd78-886x4\" (UID: \"e5ca2022-9943-4720-b501-d802020e8b4e\") " pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:14.339323 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.339300    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-886x4"]
Sep 16 13:40:14.434410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.434379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflpb\" (UniqueName: \"kubernetes.io/projected/e5ca2022-9943-4720-b501-d802020e8b4e-kube-api-access-pflpb\") pod \"authorino-operator-76b478bd78-886x4\" (UID: \"e5ca2022-9943-4720-b501-d802020e8b4e\") " pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:14.454767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.454740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflpb\" (UniqueName: \"kubernetes.io/projected/e5ca2022-9943-4720-b501-d802020e8b4e-kube-api-access-pflpb\") pod \"authorino-operator-76b478bd78-886x4\" (UID: \"e5ca2022-9943-4720-b501-d802020e8b4e\") " pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:14.634157 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.634037    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:14.730695 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.730637    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:14.856418 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.856385    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:14.919402 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.919378    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-886x4"]
Sep 16 13:40:14.921885 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:14.921855    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ca2022_9943_4720_b501_d802020e8b4e.slice/crio-634483bd6c0db348b9beb2b29c536d482107511a040b6016ca5cfee6841d2354 WatchSource:0}: Error finding container 634483bd6c0db348b9beb2b29c536d482107511a040b6016ca5cfee6841d2354: Status 404 returned error can't find the container with id 634483bd6c0db348b9beb2b29c536d482107511a040b6016ca5cfee6841d2354
Sep 16 13:40:14.958877 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:14.958819    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:15.076176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:15.076141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-886x4" event={"ID":"e5ca2022-9943-4720-b501-d802020e8b4e","Type":"ContainerStarted","Data":"634483bd6c0db348b9beb2b29c536d482107511a040b6016ca5cfee6841d2354"}
Sep 16 13:40:15.467764 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:15.467720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:15.857523 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:15.857403    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:17.138149 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:17.138070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:17.290556 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:17.290524    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:18.097820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.097777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-886x4" event={"ID":"e5ca2022-9943-4720-b501-d802020e8b4e","Type":"ContainerStarted","Data":"2a4c7a03a93cebdd2c274b2ab7bb504df4d50ff2ee7be525dde3472ba311ed0e"}
Sep 16 13:40:18.097995 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.097899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:18.118417 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.118356    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-886x4" podStartSLOduration=1.6085418790000001 podStartE2EDuration="4.118334079s" totalImagesPullingTime="2.5097922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:18.114433768 +0000 UTC m=+428.580370697" watchObservedRunningTime="2026-09-16 13:40:18.118334079 +0000 UTC m=+428.584270988"
Sep 16 13:40:18.664183 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.664149    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"]
Sep 16 13:40:18.669083 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.669058    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.671582 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.671555    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ldzgv\""
Sep 16 13:40:18.680212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.680192    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"]
Sep 16 13:40:18.773906 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.773877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84w6p\" (UniqueName: \"kubernetes.io/projected/cda18034-898f-466c-b868-61e9b9a2c270-kube-api-access-84w6p\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.774060 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.773942    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda18034-898f-466c-b868-61e9b9a2c270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.874741 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.874708    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda18034-898f-466c-b868-61e9b9a2c270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.874873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.874764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84w6p\" (UniqueName: \"kubernetes.io/projected/cda18034-898f-466c-b868-61e9b9a2c270-kube-api-access-84w6p\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.875114 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.875080    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda18034-898f-466c-b868-61e9b9a2c270-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.887622 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.887594    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84w6p\" (UniqueName: \"kubernetes.io/projected/cda18034-898f-466c-b868-61e9b9a2c270-kube-api-access-84w6p\") pod \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:18.979857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:18.979818    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:19.263369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:19.263343    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"]
Sep 16 13:40:19.265981 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:19.265956    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda18034_898f_466c_b868_61e9b9a2c270.slice/crio-adcf3fae9b6d6fa6c2fddb30d448b9b4134945274df258667b5e27da882743ff WatchSource:0}: Error finding container adcf3fae9b6d6fa6c2fddb30d448b9b4134945274df258667b5e27da882743ff: Status 404 returned error can't find the container with id adcf3fae9b6d6fa6c2fddb30d448b9b4134945274df258667b5e27da882743ff
Sep 16 13:40:19.271497 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:19.271471    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:19.894149 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:19.894083    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:20.110294 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.110256    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts" event={"ID":"cda18034-898f-466c-b868-61e9b9a2c270","Type":"ContainerStarted","Data":"adcf3fae9b6d6fa6c2fddb30d448b9b4134945274df258667b5e27da882743ff"}
Sep 16 13:40:20.845427 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.845392    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"]
Sep 16 13:40:20.849160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.849137    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:20.851302 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.851277    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-28djq\""
Sep 16 13:40:20.863261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.863237    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"]
Sep 16 13:40:20.895048 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.895021    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jn5p\" (UniqueName: \"kubernetes.io/projected/371b4f72-2fa8-4c13-a54e-39c6d7cde933-kube-api-access-4jn5p\") pod \"limitador-operator-controller-manager-74495c69f8-5mzzm\" (UID: \"371b4f72-2fa8-4c13-a54e-39c6d7cde933\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:20.996279 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:20.995834    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jn5p\" (UniqueName: \"kubernetes.io/projected/371b4f72-2fa8-4c13-a54e-39c6d7cde933-kube-api-access-4jn5p\") pod \"limitador-operator-controller-manager-74495c69f8-5mzzm\" (UID: \"371b4f72-2fa8-4c13-a54e-39c6d7cde933\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:21.008021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:21.007990    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jn5p\" (UniqueName: \"kubernetes.io/projected/371b4f72-2fa8-4c13-a54e-39c6d7cde933-kube-api-access-4jn5p\") pod \"limitador-operator-controller-manager-74495c69f8-5mzzm\" (UID: \"371b4f72-2fa8-4c13-a54e-39c6d7cde933\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:21.163900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:21.163803    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:21.458610 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:21.458566    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"]
Sep 16 13:40:21.463152 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:21.463073    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371b4f72_2fa8_4c13_a54e_39c6d7cde933.slice/crio-3685218d0a9bc9754046f272fac44f0b531018788869b0b7b9d0bb3cdda7502c WatchSource:0}: Error finding container 3685218d0a9bc9754046f272fac44f0b531018788869b0b7b9d0bb3cdda7502c: Status 404 returned error can't find the container with id 3685218d0a9bc9754046f272fac44f0b531018788869b0b7b9d0bb3cdda7502c
Sep 16 13:40:21.481565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:21.481529    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:22.124700 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.124666    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm" event={"ID":"371b4f72-2fa8-4c13-a54e-39c6d7cde933","Type":"ContainerStarted","Data":"3685218d0a9bc9754046f272fac44f0b531018788869b0b7b9d0bb3cdda7502c"}
Sep 16 13:40:22.502627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.502585    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:22.873580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.873486    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"]
Sep 16 13:40:22.877642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.877616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:22.879992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.879972    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 16 13:40:22.880143 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.879971    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9gwkg\""
Sep 16 13:40:22.889405 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.889380    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"]
Sep 16 13:40:22.916543 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:22.916518    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66bbj\" (UniqueName: \"kubernetes.io/projected/33075a5f-302c-4eb1-a901-83cab15744d4-kube-api-access-66bbj\") pod \"dns-operator-controller-manager-7b84cd7c9c-46n6f\" (UID: \"33075a5f-302c-4eb1-a901-83cab15744d4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:23.017232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.017193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66bbj\" (UniqueName: \"kubernetes.io/projected/33075a5f-302c-4eb1-a901-83cab15744d4-kube-api-access-66bbj\") pod \"dns-operator-controller-manager-7b84cd7c9c-46n6f\" (UID: \"33075a5f-302c-4eb1-a901-83cab15744d4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:23.033657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.033630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66bbj\" (UniqueName: \"kubernetes.io/projected/33075a5f-302c-4eb1-a901-83cab15744d4-kube-api-access-66bbj\") pod \"dns-operator-controller-manager-7b84cd7c9c-46n6f\" (UID: \"33075a5f-302c-4eb1-a901-83cab15744d4\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:23.133307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.133225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts" event={"ID":"cda18034-898f-466c-b868-61e9b9a2c270","Type":"ContainerStarted","Data":"0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad"}
Sep 16 13:40:23.133738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.133462    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:23.156247 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.156195    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts" podStartSLOduration=1.4023904200000001 podStartE2EDuration="5.156178874s" totalImagesPullingTime="3.753788454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:23.154801775 +0000 UTC m=+433.620738743" watchObservedRunningTime="2026-09-16 13:40:23.156178874 +0000 UTC m=+433.622115781"
Sep 16 13:40:23.192511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.192475    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:23.483897 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:23.483868    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"]
Sep 16 13:40:23.784342 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:23.784267    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33075a5f_302c_4eb1_a901_83cab15744d4.slice/crio-8337dbd4ba0a6570638ad97ac7ab28b666aed897d890f7066b5d62552117f11d WatchSource:0}: Error finding container 8337dbd4ba0a6570638ad97ac7ab28b666aed897d890f7066b5d62552117f11d: Status 404 returned error can't find the container with id 8337dbd4ba0a6570638ad97ac7ab28b666aed897d890f7066b5d62552117f11d
Sep 16 13:40:24.140483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.140375    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm" event={"ID":"371b4f72-2fa8-4c13-a54e-39c6d7cde933","Type":"ContainerStarted","Data":"a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c"}
Sep 16 13:40:24.140902 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.140569    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:24.141969 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.141939    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f" event={"ID":"33075a5f-302c-4eb1-a901-83cab15744d4","Type":"ContainerStarted","Data":"8337dbd4ba0a6570638ad97ac7ab28b666aed897d890f7066b5d62552117f11d"}
Sep 16 13:40:24.156831 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.156781    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm" podStartSLOduration=1.779176667 podStartE2EDuration="4.156765009s" totalImagesPullingTime="2.377588342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:24.15521452 +0000 UTC m=+434.621151443" watchObservedRunningTime="2026-09-16 13:40:24.156765009 +0000 UTC m=+434.622701917"
Sep 16 13:40:24.456528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.456492    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:24.512560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.512516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:24.857833 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.857745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:24.912362 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:24.912329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:25.154975 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:25.154938    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f" event={"ID":"33075a5f-302c-4eb1-a901-83cab15744d4","Type":"ContainerStarted","Data":"d71d285db62540f543c6dbe08712e328bd41543173ffed55715d1bd2b5a35116"}
Sep 16 13:40:25.155374 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:25.155194    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:25.175642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:25.175591    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f" podStartSLOduration=1.912545482 podStartE2EDuration="3.175570833s" totalImagesPullingTime="1.263025351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:25.174754751 +0000 UTC m=+435.640691684" watchObservedRunningTime="2026-09-16 13:40:25.175570833 +0000 UTC m=+435.641507740"
Sep 16 13:40:25.938877 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:25.938848    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:26.156138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:26.156106    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:29.105364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:29.105334    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-886x4"
Sep 16 13:40:30.664228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:30.664190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:30.728134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:30.728104    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:30.866080 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:30.866024    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:30.918335 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:30.918304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:31.067718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:31.067686    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:31.138940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:31.138907    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:31.708532 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:31.708493    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:31.809267 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:31.809234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:34.144800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:34.144768    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:35.157989 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.157960    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:35.423655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.423611    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"]
Sep 16 13:40:35.428874 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.428833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.443622 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.443592    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"]
Sep 16 13:40:35.529370 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.529338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblxn\" (UniqueName: \"kubernetes.io/projected/10fb805e-dff1-4445-9d5b-7000735de017-kube-api-access-bblxn\") pod \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.529572 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.529401    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10fb805e-dff1-4445-9d5b-7000735de017-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.630629 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.630590    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblxn\" (UniqueName: \"kubernetes.io/projected/10fb805e-dff1-4445-9d5b-7000735de017-kube-api-access-bblxn\") pod \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.630832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.630654    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10fb805e-dff1-4445-9d5b-7000735de017-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.631037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.631012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10fb805e-dff1-4445-9d5b-7000735de017-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.638363 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.638323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblxn\" (UniqueName: \"kubernetes.io/projected/10fb805e-dff1-4445-9d5b-7000735de017-kube-api-access-bblxn\") pod \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:35.747301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:35.747214    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:36.032957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.032934    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"]
Sep 16 13:40:36.035023 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:36.034993    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fb805e_dff1_4445_9d5b_7000735de017.slice/crio-7411e373147d789befe283ac969168dd92fdfe3333b245c4b378d13dc33e8345 WatchSource:0}: Error finding container 7411e373147d789befe283ac969168dd92fdfe3333b245c4b378d13dc33e8345: Status 404 returned error can't find the container with id 7411e373147d789befe283ac969168dd92fdfe3333b245c4b378d13dc33e8345
Sep 16 13:40:36.163567 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.163539    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-46n6f"
Sep 16 13:40:36.217816 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.217782    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4" event={"ID":"10fb805e-dff1-4445-9d5b-7000735de017","Type":"ContainerStarted","Data":"448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d"}
Sep 16 13:40:36.217977 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.217942    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:36.217977 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.217963    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4" event={"ID":"10fb805e-dff1-4445-9d5b-7000735de017","Type":"ContainerStarted","Data":"7411e373147d789befe283ac969168dd92fdfe3333b245c4b378d13dc33e8345"}
Sep 16 13:40:36.240518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.240458    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4" podStartSLOduration=1.240442687 podStartE2EDuration="1.240442687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:36.237132927 +0000 UTC m=+446.703069858" watchObservedRunningTime="2026-09-16 13:40:36.240442687 +0000 UTC m=+446.706379607"
Sep 16 13:40:36.339630 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.339546    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"]
Sep 16 13:40:36.349915 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.349882    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"]
Sep 16 13:40:36.367574 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.367543    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"]
Sep 16 13:40:36.368072 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.368052    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10fb805e-dff1-4445-9d5b-7000735de017" containerName="manager"
Sep 16 13:40:36.368072 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.368075    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="10fb805e-dff1-4445-9d5b-7000735de017" containerName="manager"
Sep 16 13:40:36.368262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.368192    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="10fb805e-dff1-4445-9d5b-7000735de017" containerName="manager"
Sep 16 13:40:36.371401 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.371381    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"]
Sep 16 13:40:36.371527 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.371504    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.371615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.371591    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts" podUID="cda18034-898f-466c-b868-61e9b9a2c270" containerName="manager" containerID="cri-o://0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad" gracePeriod=2
Sep 16 13:40:36.379480 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.379451    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"]
Sep 16 13:40:36.398733 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.398703    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"]
Sep 16 13:40:36.401826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.401802    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"]
Sep 16 13:40:36.402251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.402236    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda18034-898f-466c-b868-61e9b9a2c270" containerName="manager"
Sep 16 13:40:36.402306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.402253    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda18034-898f-466c-b868-61e9b9a2c270" containerName="manager"
Sep 16 13:40:36.402343 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.402307    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda18034-898f-466c-b868-61e9b9a2c270" containerName="manager"
Sep 16 13:40:36.405430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.405409    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"]
Sep 16 13:40:36.405606 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.405588    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.405683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.405614    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" containerName="manager" containerID="cri-o://a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c" gracePeriod=2
Sep 16 13:40:36.412199 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.412163    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:36.416295 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.416261    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"]
Sep 16 13:40:36.419143 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.419120    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"]
Sep 16 13:40:36.431541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.431515    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"]
Sep 16 13:40:36.432072 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.432055    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" containerName="manager"
Sep 16 13:40:36.432150 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.432078    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" containerName="manager"
Sep 16 13:40:36.432232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.432219    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" containerName="manager"
Sep 16 13:40:36.435652 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.435631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:36.437218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.437116    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-shk8x\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.437218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.437169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz88d\" (UniqueName: \"kubernetes.io/projected/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-kube-api-access-xz88d\") pod \"kuadrant-operator-controller-manager-5f685c496f-shk8x\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.447292 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.447268    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"]
Sep 16 13:40:36.447587 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.447554    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:36.449070 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.449048    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5mzzm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:36.476222 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.476193    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:36.538792 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.538733    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fr9z2\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.538792 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.538784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj774\" (UniqueName: \"kubernetes.io/projected/ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b-kube-api-access-zj774\") pod \"limitador-operator-controller-manager-74495c69f8-kc4n6\" (UID: \"ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:36.539038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.538893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88fj\" (UniqueName: \"kubernetes.io/projected/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-kube-api-access-b88fj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fr9z2\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.539038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.538962    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-shk8x\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.539038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.538994    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz88d\" (UniqueName: \"kubernetes.io/projected/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-kube-api-access-xz88d\") pod \"kuadrant-operator-controller-manager-5f685c496f-shk8x\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.539403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.539366    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-shk8x\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.551443 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.551416    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz88d\" (UniqueName: \"kubernetes.io/projected/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-kube-api-access-xz88d\") pod \"kuadrant-operator-controller-manager-5f685c496f-shk8x\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.634440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.634415    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:36.636367 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.636339    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:36.637705 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.637680    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5mzzm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:36.639488 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.639466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fr9z2\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.639591 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.639505    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj774\" (UniqueName: \"kubernetes.io/projected/ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b-kube-api-access-zj774\") pod \"limitador-operator-controller-manager-74495c69f8-kc4n6\" (UID: \"ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:36.639591 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.639555    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88fj\" (UniqueName: \"kubernetes.io/projected/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-kube-api-access-b88fj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fr9z2\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.639885 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.639865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fr9z2\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.641934 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.641915    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:36.643543 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.643524    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:36.644957 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:36.644898    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5mzzm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:36.648973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.648951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj774\" (UniqueName: \"kubernetes.io/projected/ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b-kube-api-access-zj774\") pod \"limitador-operator-controller-manager-74495c69f8-kc4n6\" (UID: \"ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:36.649069 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.648951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88fj\" (UniqueName: \"kubernetes.io/projected/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-kube-api-access-b88fj\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fr9z2\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.685691 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.685668    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:36.741028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.740569    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda18034-898f-466c-b868-61e9b9a2c270-extensions-socket-volume\") pod \"cda18034-898f-466c-b868-61e9b9a2c270\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") "
Sep 16 13:40:36.741028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.740617    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84w6p\" (UniqueName: \"kubernetes.io/projected/cda18034-898f-466c-b868-61e9b9a2c270-kube-api-access-84w6p\") pod \"cda18034-898f-466c-b868-61e9b9a2c270\" (UID: \"cda18034-898f-466c-b868-61e9b9a2c270\") "
Sep 16 13:40:36.741028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.740670    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jn5p\" (UniqueName: \"kubernetes.io/projected/371b4f72-2fa8-4c13-a54e-39c6d7cde933-kube-api-access-4jn5p\") pod \"371b4f72-2fa8-4c13-a54e-39c6d7cde933\" (UID: \"371b4f72-2fa8-4c13-a54e-39c6d7cde933\") "
Sep 16 13:40:36.741028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.740992    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda18034-898f-466c-b868-61e9b9a2c270-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cda18034-898f-466c-b868-61e9b9a2c270" (UID: "cda18034-898f-466c-b868-61e9b9a2c270"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:36.744424 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.744373    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371b4f72-2fa8-4c13-a54e-39c6d7cde933-kube-api-access-4jn5p" (OuterVolumeSpecName: "kube-api-access-4jn5p") pod "371b4f72-2fa8-4c13-a54e-39c6d7cde933" (UID: "371b4f72-2fa8-4c13-a54e-39c6d7cde933"). InnerVolumeSpecName "kube-api-access-4jn5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:36.744976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.744948    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda18034-898f-466c-b868-61e9b9a2c270-kube-api-access-84w6p" (OuterVolumeSpecName: "kube-api-access-84w6p") pod "cda18034-898f-466c-b868-61e9b9a2c270" (UID: "cda18034-898f-466c-b868-61e9b9a2c270"). InnerVolumeSpecName "kube-api-access-84w6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:36.787365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.787244    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:36.795663 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.795635    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:36.841774 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.841525    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jn5p\" (UniqueName: \"kubernetes.io/projected/371b4f72-2fa8-4c13-a54e-39c6d7cde933-kube-api-access-4jn5p\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:36.841774 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.841563    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cda18034-898f-466c-b868-61e9b9a2c270-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:36.841774 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.841579    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84w6p\" (UniqueName: \"kubernetes.io/projected/cda18034-898f-466c-b868-61e9b9a2c270-kube-api-access-84w6p\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:36.966905 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:36.966874    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"]
Sep 16 13:40:36.968229 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:36.968207    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe2c99f_ae6c_4500_bce9_bdeb9db8363a.slice/crio-31a3443afd6bd802d7c57e5ec646bff014ab7a6acf49b732bee2fb061223684f WatchSource:0}: Error finding container 31a3443afd6bd802d7c57e5ec646bff014ab7a6acf49b732bee2fb061223684f: Status 404 returned error can't find the container with id 31a3443afd6bd802d7c57e5ec646bff014ab7a6acf49b732bee2fb061223684f
Sep 16 13:40:37.076822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.076784    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"]
Sep 16 13:40:37.079204 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.079183    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"]
Sep 16 13:40:37.079844 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:37.079816    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8e699a_d72e_4db2_b82f_e6fe1ff3651f.slice/crio-bb3c5a8296668429a03bbcb8ca7424e242925f743b212c8426294d2b86391638 WatchSource:0}: Error finding container bb3c5a8296668429a03bbcb8ca7424e242925f743b212c8426294d2b86391638: Status 404 returned error can't find the container with id bb3c5a8296668429a03bbcb8ca7424e242925f743b212c8426294d2b86391638
Sep 16 13:40:37.080778 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:40:37.080753    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff61ebd2_8da1_4fe4_98c6_261fd4c0c31b.slice/crio-c8c08f898601b5f401c3e7da6c9e23c7524dd92fc2ed047a55674f7868f95710 WatchSource:0}: Error finding container c8c08f898601b5f401c3e7da6c9e23c7524dd92fc2ed047a55674f7868f95710: Status 404 returned error can't find the container with id c8c08f898601b5f401c3e7da6c9e23c7524dd92fc2ed047a55674f7868f95710
Sep 16 13:40:37.106082 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.106058    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:37.176547 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.176516    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-57785d5cf8-m929f" podUID="7a1bd368-879b-49cb-a76a-7ff7c847d76a" containerName="console" containerID="cri-o://ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d" gracePeriod=15
Sep 16 13:40:37.207491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.207454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:37.227589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.227548    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x" event={"ID":"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a","Type":"ContainerStarted","Data":"bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f"}
Sep 16 13:40:37.227763 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.227655    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:37.227763 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.227681    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x" event={"ID":"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a","Type":"ContainerStarted","Data":"31a3443afd6bd802d7c57e5ec646bff014ab7a6acf49b732bee2fb061223684f"}
Sep 16 13:40:37.229234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.229146    2556 generic.go:356] "Generic (PLEG): container finished" podID="cda18034-898f-466c-b868-61e9b9a2c270" containerID="0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad" exitCode=0
Sep 16 13:40:37.229435 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.229238    2556 scope.go:117] "RemoveContainer" containerID="0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad"
Sep 16 13:40:37.229505 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.229488    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5mzzm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:37.229625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.229606    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:37.231115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.231071    2556 generic.go:356] "Generic (PLEG): container finished" podID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" containerID="a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c" exitCode=0
Sep 16 13:40:37.231189 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.231140    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:37.232891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.232855    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6" event={"ID":"ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b","Type":"ContainerStarted","Data":"a64eee603a21ed741d887ee1cdb149c83474d434d5f3c8a728aed4d699780d72"}
Sep 16 13:40:37.232994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.232905    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6" event={"ID":"ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b","Type":"ContainerStarted","Data":"c8c08f898601b5f401c3e7da6c9e23c7524dd92fc2ed047a55674f7868f95710"}
Sep 16 13:40:37.232994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.232934    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:37.234751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.234722    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2" event={"ID":"de8e699a-d72e-4db2-b82f-e6fe1ff3651f","Type":"ContainerStarted","Data":"6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df"}
Sep 16 13:40:37.234832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.234779    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2" event={"ID":"de8e699a-d72e-4db2-b82f-e6fe1ff3651f","Type":"ContainerStarted","Data":"bb3c5a8296668429a03bbcb8ca7424e242925f743b212c8426294d2b86391638"}
Sep 16 13:40:37.234832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.234807    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:37.234996 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.234979    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4" container="manager"
Sep 16 13:40:37.248449 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.248414    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x" podStartSLOduration=1.248400026 podStartE2EDuration="1.248400026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:37.247067355 +0000 UTC m=+447.713004264" watchObservedRunningTime="2026-09-16 13:40:37.248400026 +0000 UTC m=+447.714336934"
Sep 16 13:40:37.248638 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.248617    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:37.250212 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.250191    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:37.251542 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.251525    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="10fb805e-dff1-4445-9d5b-7000735de017" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:37.252886 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.252867    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5mzzm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:37.271987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.271948    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6" podStartSLOduration=1.271935259 podStartE2EDuration="1.271935259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:37.270176823 +0000 UTC m=+447.736113763" watchObservedRunningTime="2026-09-16 13:40:37.271935259 +0000 UTC m=+447.737872170"
Sep 16 13:40:37.290298 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.290269    2556 scope.go:117] "RemoveContainer" containerID="0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad"
Sep 16 13:40:37.290709 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.290680    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad\": container with ID starting with 0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad not found: ID does not exist" containerID="0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad"
Sep 16 13:40:37.290826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.290722    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad"} err="failed to get container status \"0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad\": rpc error: code = NotFound desc = could not find container \"0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad\": container with ID starting with 0ea8e5912da10a29fa5e3200877b9a07ad8f220254aad3b841ebfbc8dd748aad not found: ID does not exist"
Sep 16 13:40:37.290826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.290749    2556 scope.go:117] "RemoveContainer" containerID="a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c"
Sep 16 13:40:37.292310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.292270    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2" podStartSLOduration=1.292252238 podStartE2EDuration="1.292252238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:40:37.288263861 +0000 UTC m=+447.754200770" watchObservedRunningTime="2026-09-16 13:40:37.292252238 +0000 UTC m=+447.758189147"
Sep 16 13:40:37.302851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.302821    2556 scope.go:117] "RemoveContainer" containerID="a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c"
Sep 16 13:40:37.303350 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.303248    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c\": container with ID starting with a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c not found: ID does not exist" containerID="a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c"
Sep 16 13:40:37.303350 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.303289    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c"} err="failed to get container status \"a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c\": rpc error: code = NotFound desc = could not find container \"a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c\": container with ID starting with a18e18629c6c10aebd25e2760ceb258b9af32fe3db5a183da598d744a5dee40c not found: ID does not exist"
Sep 16 13:40:37.416932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.416902    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:40:37.419191 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.419164    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-zdjts\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="cda18034-898f-466c-b868-61e9b9a2c270" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-zdjts"
Sep 16 13:40:37.462033 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.461994    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="10fb805e-dff1-4445-9d5b-7000735de017" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:37.464004 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:37.463969    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5mzzm\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5mzzm"
Sep 16 13:40:37.548561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548528    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-config\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.548561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548567    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-oauth-config\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.548796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548591    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc64h\" (UniqueName: \"kubernetes.io/projected/7a1bd368-879b-49cb-a76a-7ff7c847d76a-kube-api-access-jc64h\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.548796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548612    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-trusted-ca-bundle\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.548796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548681    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-service-ca\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.548796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548716    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-serving-cert\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.548796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548770    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-oauth-serving-cert\") pod \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\" (UID: \"7a1bd368-879b-49cb-a76a-7ff7c847d76a\") "
Sep 16 13:40:37.549038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.548969    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-config" (OuterVolumeSpecName: "console-config") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:40:37.549199 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.549160    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:40:37.549199 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.549172    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-service-ca" (OuterVolumeSpecName: "service-ca") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:40:37.549577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.549290    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:40:37.551000 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.550974    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:40:37.551000 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.550987    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:40:37.551152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.551020    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1bd368-879b-49cb-a76a-7ff7c847d76a-kube-api-access-jc64h" (OuterVolumeSpecName: "kube-api-access-jc64h") pod "7a1bd368-879b-49cb-a76a-7ff7c847d76a" (UID: "7a1bd368-879b-49cb-a76a-7ff7c847d76a"). InnerVolumeSpecName "kube-api-access-jc64h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:37.649470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649435    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:37.649470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649468    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-serving-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:37.649682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649484    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-oauth-serving-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:37.649682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649512    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:37.649682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649524    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1bd368-879b-49cb-a76a-7ff7c847d76a-console-oauth-config\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:37.649682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649534    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc64h\" (UniqueName: \"kubernetes.io/projected/7a1bd368-879b-49cb-a76a-7ff7c847d76a-kube-api-access-jc64h\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:37.649682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:37.649547    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1bd368-879b-49cb-a76a-7ff7c847d76a-trusted-ca-bundle\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:38.059774 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.059739    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371b4f72-2fa8-4c13-a54e-39c6d7cde933" path="/var/lib/kubelet/pods/371b4f72-2fa8-4c13-a54e-39c6d7cde933/volumes"
Sep 16 13:40:38.060129 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.060116    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cda18034-898f-466c-b868-61e9b9a2c270" path="/var/lib/kubelet/pods/cda18034-898f-466c-b868-61e9b9a2c270/volumes"
Sep 16 13:40:38.244661 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.244598    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4" podUID="10fb805e-dff1-4445-9d5b-7000735de017" containerName="manager" containerID="cri-o://448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d" gracePeriod=2
Sep 16 13:40:38.246014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.245987    2556 generic.go:356] "Generic (PLEG): container finished" podID="7a1bd368-879b-49cb-a76a-7ff7c847d76a" containerID="ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d" exitCode=0
Sep 16 13:40:38.246128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.246020    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57785d5cf8-m929f"
Sep 16 13:40:38.246128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.246051    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57785d5cf8-m929f" event={"ID":"7a1bd368-879b-49cb-a76a-7ff7c847d76a","Type":"ContainerDied","Data":"ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d"}
Sep 16 13:40:38.246128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.246119    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57785d5cf8-m929f" event={"ID":"7a1bd368-879b-49cb-a76a-7ff7c847d76a","Type":"ContainerDied","Data":"b4d6d95d323c3f403fa812d57e937603c89315d7e350f6054dc83dd8098d4e41"}
Sep 16 13:40:38.246248 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.246137    2556 scope.go:117] "RemoveContainer" containerID="ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d"
Sep 16 13:40:38.255946 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.255926    2556 scope.go:117] "RemoveContainer" containerID="ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d"
Sep 16 13:40:38.256235 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:38.256216    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d\": container with ID starting with ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d not found: ID does not exist" containerID="ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d"
Sep 16 13:40:38.256310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.256245    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d"} err="failed to get container status \"ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d\": rpc error: code = NotFound desc = could not find container \"ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d\": container with ID starting with ade31ca349c53903013d43e23650bfc04b4f0852d7ec24270521b0cf7d36c09d not found: ID does not exist"
Sep 16 13:40:38.263799 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.263780    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57785d5cf8-m929f"]
Sep 16 13:40:38.269133 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.269114    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57785d5cf8-m929f"]
Sep 16 13:40:38.493473 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.493446    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:38.557031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.556995    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bblxn\" (UniqueName: \"kubernetes.io/projected/10fb805e-dff1-4445-9d5b-7000735de017-kube-api-access-bblxn\") pod \"10fb805e-dff1-4445-9d5b-7000735de017\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") "
Sep 16 13:40:38.557210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.557072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10fb805e-dff1-4445-9d5b-7000735de017-extensions-socket-volume\") pod \"10fb805e-dff1-4445-9d5b-7000735de017\" (UID: \"10fb805e-dff1-4445-9d5b-7000735de017\") "
Sep 16 13:40:38.557446 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.557423    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10fb805e-dff1-4445-9d5b-7000735de017-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "10fb805e-dff1-4445-9d5b-7000735de017" (UID: "10fb805e-dff1-4445-9d5b-7000735de017"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:38.559256 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.559236    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fb805e-dff1-4445-9d5b-7000735de017-kube-api-access-bblxn" (OuterVolumeSpecName: "kube-api-access-bblxn") pod "10fb805e-dff1-4445-9d5b-7000735de017" (UID: "10fb805e-dff1-4445-9d5b-7000735de017"). InnerVolumeSpecName "kube-api-access-bblxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:38.657771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.657686    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/10fb805e-dff1-4445-9d5b-7000735de017-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:38.657771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:38.657713    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblxn\" (UniqueName: \"kubernetes.io/projected/10fb805e-dff1-4445-9d5b-7000735de017-kube-api-access-bblxn\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:39.254419 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.254385    2556 generic.go:356] "Generic (PLEG): container finished" podID="10fb805e-dff1-4445-9d5b-7000735de017" containerID="448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d" exitCode=0
Sep 16 13:40:39.254857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.254439    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:39.254857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.254484    2556 scope.go:117] "RemoveContainer" containerID="448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d"
Sep 16 13:40:39.256330 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:39.256300    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="10fb805e-dff1-4445-9d5b-7000735de017" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:39.263942 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.263925    2556 scope.go:117] "RemoveContainer" containerID="448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d"
Sep 16 13:40:39.264240 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:39.264222    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d\": container with ID starting with 448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d not found: ID does not exist" containerID="448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d"
Sep 16 13:40:39.264303 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.264247    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d"} err="failed to get container status \"448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d\": rpc error: code = NotFound desc = could not find container \"448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d\": container with ID starting with 448d4ace54deaf52435e99f7220957ce23415e6fc5318a73215a36f84047cf1d not found: ID does not exist"
Sep 16 13:40:39.264609 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:39.264589    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="10fb805e-dff1-4445-9d5b-7000735de017" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:39.570447 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.570359    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:39.670231 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:39.670196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:40.061520 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:40.061485    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-djcs4\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="10fb805e-dff1-4445-9d5b-7000735de017" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-djcs4"
Sep 16 13:40:40.064317 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:40.064289    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10fb805e-dff1-4445-9d5b-7000735de017" path="/var/lib/kubelet/pods/10fb805e-dff1-4445-9d5b-7000735de017/volumes"
Sep 16 13:40:40.064742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:40.064724    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1bd368-879b-49cb-a76a-7ff7c847d76a" path="/var/lib/kubelet/pods/7a1bd368-879b-49cb-a76a-7ff7c847d76a/volumes"
Sep 16 13:40:45.278655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:45.278612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:45.333826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:45.333791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:45.883969 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:45.883937    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:45.945057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:45.945020    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:48.248635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.248601    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:48.248988 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.248734    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:40:48.249034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.249017    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kc4n6"
Sep 16 13:40:48.355162 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.355134    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"]
Sep 16 13:40:48.355389 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.355355    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2" podUID="de8e699a-d72e-4db2-b82f-e6fe1ff3651f" containerName="manager" containerID="cri-o://6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df" gracePeriod=10
Sep 16 13:40:48.621845 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.621823    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:48.745261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.745226    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-extensions-socket-volume\") pod \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") "
Sep 16 13:40:48.745480 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.745295    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b88fj\" (UniqueName: \"kubernetes.io/projected/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-kube-api-access-b88fj\") pod \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\" (UID: \"de8e699a-d72e-4db2-b82f-e6fe1ff3651f\") "
Sep 16 13:40:48.745600 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.745575    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "de8e699a-d72e-4db2-b82f-e6fe1ff3651f" (UID: "de8e699a-d72e-4db2-b82f-e6fe1ff3651f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:40:48.747466 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.747444    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-kube-api-access-b88fj" (OuterVolumeSpecName: "kube-api-access-b88fj") pod "de8e699a-d72e-4db2-b82f-e6fe1ff3651f" (UID: "de8e699a-d72e-4db2-b82f-e6fe1ff3651f"). InnerVolumeSpecName "kube-api-access-b88fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:40:48.846550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.846489    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:48.846550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:48.846513    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88fj\" (UniqueName: \"kubernetes.io/projected/de8e699a-d72e-4db2-b82f-e6fe1ff3651f-kube-api-access-b88fj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:40:49.317643 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.317608    2556 generic.go:356] "Generic (PLEG): container finished" podID="de8e699a-d72e-4db2-b82f-e6fe1ff3651f" containerID="6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df" exitCode=0
Sep 16 13:40:49.317643 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.317644    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2" event={"ID":"de8e699a-d72e-4db2-b82f-e6fe1ff3651f","Type":"ContainerDied","Data":"6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df"}
Sep 16 13:40:49.318177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.317681    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"
Sep 16 13:40:49.318177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.317698    2556 scope.go:117] "RemoveContainer" containerID="6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df"
Sep 16 13:40:49.318177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.317680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2" event={"ID":"de8e699a-d72e-4db2-b82f-e6fe1ff3651f","Type":"ContainerDied","Data":"bb3c5a8296668429a03bbcb8ca7424e242925f743b212c8426294d2b86391638"}
Sep 16 13:40:49.326600 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.326583    2556 scope.go:117] "RemoveContainer" containerID="6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df"
Sep 16 13:40:49.326874 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:40:49.326852    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df\": container with ID starting with 6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df not found: ID does not exist" containerID="6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df"
Sep 16 13:40:49.326919 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.326888    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df"} err="failed to get container status \"6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df\": rpc error: code = NotFound desc = could not find container \"6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df\": container with ID starting with 6507d1e7519aaa9a4e790626b9c168ea0971ca8dbb450ea2a9a882ff7a4ef5df not found: ID does not exist"
Sep 16 13:40:49.375545 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.375516    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"]
Sep 16 13:40:49.380794 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:49.380775    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fr9z2"]
Sep 16 13:40:50.058694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:50.058659    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de8e699a-d72e-4db2-b82f-e6fe1ff3651f" path="/var/lib/kubelet/pods/de8e699a-d72e-4db2-b82f-e6fe1ff3651f/volumes"
Sep 16 13:40:50.285326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:50.285291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:50.396587 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:50.396507    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:50.686575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:50.686550    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:50.786613 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:50.786588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:51.085418 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:51.085329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:51.192208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:51.192181    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:57.107363 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:57.107332    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:57.207761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:57.207728    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:57.892368 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:57.892333    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:59.571068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:59.571028    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:40:59.671257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:40:59.671221    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:04.485142 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.485080    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"]
Sep 16 13:41:04.485649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.485350    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x" podUID="ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" containerName="manager" containerID="cri-o://bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f" gracePeriod=10
Sep 16 13:41:04.703431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.703394    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"]
Sep 16 13:41:04.704081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.704054    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8e699a-d72e-4db2-b82f-e6fe1ff3651f" containerName="manager"
Sep 16 13:41:04.704265 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.704250    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8e699a-d72e-4db2-b82f-e6fe1ff3651f" containerName="manager"
Sep 16 13:41:04.704408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.704387    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1bd368-879b-49cb-a76a-7ff7c847d76a" containerName="console"
Sep 16 13:41:04.704408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.704409    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1bd368-879b-49cb-a76a-7ff7c847d76a" containerName="console"
Sep 16 13:41:04.704541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.704528    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="de8e699a-d72e-4db2-b82f-e6fe1ff3651f" containerName="manager"
Sep 16 13:41:04.704577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.704542    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1bd368-879b-49cb-a76a-7ff7c847d76a" containerName="console"
Sep 16 13:41:04.707434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.707413    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.709685 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.709667    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p8jqm\""
Sep 16 13:41:04.709685 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.709677    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 13:41:04.720616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.720591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"]
Sep 16 13:41:04.741151 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.741070    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:04.747798 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.747779    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:41:04.785609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785580    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz88d\" (UniqueName: \"kubernetes.io/projected/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-kube-api-access-xz88d\") pod \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") "
Sep 16 13:41:04.785796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-istio-envoy\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.785796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785688    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20b64c-0972-4684-9fe0-c0f769331b52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.785796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxnn\" (UniqueName: \"kubernetes.io/projected/af20b64c-0972-4684-9fe0-c0f769331b52-kube-api-access-6pxnn\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.785796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-workload-certs\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.786016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-istio-data\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.786016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785843    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-credential-socket\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.786016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785910    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-workload-socket\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.786016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20b64c-0972-4684-9fe0-c0f769331b52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.786016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.785946    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20b64c-0972-4684-9fe0-c0f769331b52-istio-token\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.787918 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.787889    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-kube-api-access-xz88d" (OuterVolumeSpecName: "kube-api-access-xz88d") pod "ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" (UID: "ebe2c99f-ae6c-4500-bce9-bdeb9db8363a"). InnerVolumeSpecName "kube-api-access-xz88d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:41:04.886689 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.886648    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-extensions-socket-volume\") pod \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\" (UID: \"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a\") "
Sep 16 13:41:04.886893 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.886813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-credential-socket\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.886893 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.886885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-workload-socket\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.886912    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20b64c-0972-4684-9fe0-c0f769331b52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.886942    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20b64c-0972-4684-9fe0-c0f769331b52-istio-token\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.886984    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-istio-envoy\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20b64c-0972-4684-9fe0-c0f769331b52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxnn\" (UniqueName: \"kubernetes.io/projected/af20b64c-0972-4684-9fe0-c0f769331b52-kube-api-access-6pxnn\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887146    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" (UID: "ebe2c99f-ae6c-4500-bce9-bdeb9db8363a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:41:04.887209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-workload-certs\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887186    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-istio-data\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887236    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz88d\" (UniqueName: \"kubernetes.io/projected/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-kube-api-access-xz88d\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:41:04.887451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887250    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a-extensions-socket-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:41:04.887451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887283    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-credential-socket\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-workload-socket\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887537    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-istio-data\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887607    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-workload-certs\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.887842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.887824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af20b64c-0972-4684-9fe0-c0f769331b52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.889564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.889544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af20b64c-0972-4684-9fe0-c0f769331b52-istio-envoy\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.889670 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.889639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af20b64c-0972-4684-9fe0-c0f769331b52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.894960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.894933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af20b64c-0972-4684-9fe0-c0f769331b52-istio-token\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:04.895125 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:04.895058    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxnn\" (UniqueName: \"kubernetes.io/projected/af20b64c-0972-4684-9fe0-c0f769331b52-kube-api-access-6pxnn\") pod \"maas-default-gateway-openshift-default-df6776dd6-gkxhg\" (UID: \"af20b64c-0972-4684-9fe0-c0f769331b52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:05.020909 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.020805    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:05.021219 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.021199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:05.295322 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.295297    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"]
Sep 16 13:41:05.296257 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:41:05.296231    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf20b64c_0972_4684_9fe0_c0f769331b52.slice/crio-b769b54d54c1e4ed773b413422e42732ecc484c71b046e3d7aec5e650211daf9 WatchSource:0}: Error finding container b769b54d54c1e4ed773b413422e42732ecc484c71b046e3d7aec5e650211daf9: Status 404 returned error can't find the container with id b769b54d54c1e4ed773b413422e42732ecc484c71b046e3d7aec5e650211daf9
Sep 16 13:41:05.408631 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.408594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" event={"ID":"af20b64c-0972-4684-9fe0-c0f769331b52","Type":"ContainerStarted","Data":"b769b54d54c1e4ed773b413422e42732ecc484c71b046e3d7aec5e650211daf9"}
Sep 16 13:41:05.410187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.410160    2556 generic.go:356] "Generic (PLEG): container finished" podID="ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" containerID="bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f" exitCode=0
Sep 16 13:41:05.410327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.410196    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x" event={"ID":"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a","Type":"ContainerDied","Data":"bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f"}
Sep 16 13:41:05.410327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.410208    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"
Sep 16 13:41:05.410327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.410231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x" event={"ID":"ebe2c99f-ae6c-4500-bce9-bdeb9db8363a","Type":"ContainerDied","Data":"31a3443afd6bd802d7c57e5ec646bff014ab7a6acf49b732bee2fb061223684f"}
Sep 16 13:41:05.410327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.410253    2556 scope.go:117] "RemoveContainer" containerID="bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f"
Sep 16 13:41:05.420176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.420159    2556 scope.go:117] "RemoveContainer" containerID="bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f"
Sep 16 13:41:05.420442 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:41:05.420422    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f\": container with ID starting with bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f not found: ID does not exist" containerID="bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f"
Sep 16 13:41:05.420491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.420452    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f"} err="failed to get container status \"bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f\": rpc error: code = NotFound desc = could not find container \"bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f\": container with ID starting with bd48cc89a1dbd7ad7a360bbc14727e4bf3850934f8b70cb39019817ddafe644f not found: ID does not exist"
Sep 16 13:41:05.432674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.432646    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"]
Sep 16 13:41:05.440490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:05.440458    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-shk8x"]
Sep 16 13:41:06.058863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:06.058830    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" path="/var/lib/kubelet/pods/ebe2c99f-ae6c-4500-bce9-bdeb9db8363a/volumes"
Sep 16 13:41:14.690346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:14.690306    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:41:14.690642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:14.690382    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:41:14.690642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:14.690411    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:41:15.468160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:15.468123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" event={"ID":"af20b64c-0972-4684-9fe0-c0f769331b52","Type":"ContainerStarted","Data":"fe3a05b484b64462edd35ff52a39bc5830d523336ff7c738ee0d15da3b2b8bef"}
Sep 16 13:41:15.490081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:15.490024    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" podStartSLOduration=2.098328652 podStartE2EDuration="11.490008294s" totalImagesPullingTime="9.391679642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:15.487269525 +0000 UTC m=+485.953206456" watchObservedRunningTime="2026-09-16 13:41:15.490008294 +0000 UTC m=+485.955945202"
Sep 16 13:41:16.021691 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:16.021645    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:16.026591 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:16.026564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:16.473681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:16.473647    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:16.474584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:16.474564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:41:24.614810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.614776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:24.622656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.622632    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:24.623031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.623018    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" containerName="manager"
Sep 16 13:41:24.623080 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.623032    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" containerName="manager"
Sep 16 13:41:24.623137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.623103    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe2c99f-ae6c-4500-bce9-bdeb9db8363a" containerName="manager"
Sep 16 13:41:24.626288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.626273    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.628168 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.628150    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fz48k\""
Sep 16 13:41:24.628283 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.628225    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 13:41:24.633625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.633603    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:24.717770 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.717742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:24.723854 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.723826    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:24.785309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.785272    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9d9\" (UniqueName: \"kubernetes.io/projected/81f3cb95-9aa8-422e-a513-a57d0109d193-kube-api-access-8r9d9\") pod \"limitador-limitador-8588749977-44778\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") " pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.785460 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.785357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f3cb95-9aa8-422e-a513-a57d0109d193-config-file\") pod \"limitador-limitador-8588749977-44778\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") " pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.818476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.818443    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:24.872433 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.872359    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:41:24.875959 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.875942    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:24.882757 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.882737    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:41:24.886178 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.886155    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9d9\" (UniqueName: \"kubernetes.io/projected/81f3cb95-9aa8-422e-a513-a57d0109d193-kube-api-access-8r9d9\") pod \"limitador-limitador-8588749977-44778\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") " pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.886290 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.886208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f3cb95-9aa8-422e-a513-a57d0109d193-config-file\") pod \"limitador-limitador-8588749977-44778\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") " pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.886985 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.886963    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f3cb95-9aa8-422e-a513-a57d0109d193-config-file\") pod \"limitador-limitador-8588749977-44778\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") " pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.894548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.894530    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9d9\" (UniqueName: \"kubernetes.io/projected/81f3cb95-9aa8-422e-a513-a57d0109d193-kube-api-access-8r9d9\") pod \"limitador-limitador-8588749977-44778\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") " pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.911284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.911255    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:41:24.937848 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.937821    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:24.987633 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.987415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebfc6e40-8d5b-402f-82cb-da81f6bd4343-config-file\") pod \"limitador-limitador-84994f7ffc-47spw\" (UID: \"ebfc6e40-8d5b-402f-82cb-da81f6bd4343\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:24.987633 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:24.987481    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjp2\" (UniqueName: \"kubernetes.io/projected/ebfc6e40-8d5b-402f-82cb-da81f6bd4343-kube-api-access-gbjp2\") pod \"limitador-limitador-84994f7ffc-47spw\" (UID: \"ebfc6e40-8d5b-402f-82cb-da81f6bd4343\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:25.087962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.087929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjp2\" (UniqueName: \"kubernetes.io/projected/ebfc6e40-8d5b-402f-82cb-da81f6bd4343-kube-api-access-gbjp2\") pod \"limitador-limitador-84994f7ffc-47spw\" (UID: \"ebfc6e40-8d5b-402f-82cb-da81f6bd4343\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:25.088144 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.088013    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebfc6e40-8d5b-402f-82cb-da81f6bd4343-config-file\") pod \"limitador-limitador-84994f7ffc-47spw\" (UID: \"ebfc6e40-8d5b-402f-82cb-da81f6bd4343\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:25.088620 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.088601    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ebfc6e40-8d5b-402f-82cb-da81f6bd4343-config-file\") pod \"limitador-limitador-84994f7ffc-47spw\" (UID: \"ebfc6e40-8d5b-402f-82cb-da81f6bd4343\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:25.095499 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.095475    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjp2\" (UniqueName: \"kubernetes.io/projected/ebfc6e40-8d5b-402f-82cb-da81f6bd4343-kube-api-access-gbjp2\") pod \"limitador-limitador-84994f7ffc-47spw\" (UID: \"ebfc6e40-8d5b-402f-82cb-da81f6bd4343\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:25.187310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.187281    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:25.208720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:41:25.208696    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f3cb95_9aa8_422e_a513_a57d0109d193.slice/crio-c19d62b313bbaabf8c7c6ea382c87b58f4691641e3b19e29f12a8fdffec1da56 WatchSource:0}: Error finding container c19d62b313bbaabf8c7c6ea382c87b58f4691641e3b19e29f12a8fdffec1da56: Status 404 returned error can't find the container with id c19d62b313bbaabf8c7c6ea382c87b58f4691641e3b19e29f12a8fdffec1da56
Sep 16 13:41:25.209262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.209236    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:25.428882 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.428849    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:25.461665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.461643    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:41:25.462864 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:41:25.462830    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfc6e40_8d5b_402f_82cb_da81f6bd4343.slice/crio-720cffc30d9d0f5be617c4f45594d247ee2dfc01e07f74e01830da0dea0126d6 WatchSource:0}: Error finding container 720cffc30d9d0f5be617c4f45594d247ee2dfc01e07f74e01830da0dea0126d6: Status 404 returned error can't find the container with id 720cffc30d9d0f5be617c4f45594d247ee2dfc01e07f74e01830da0dea0126d6
Sep 16 13:41:25.505319 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.505283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:25.524160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.524126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw" event={"ID":"ebfc6e40-8d5b-402f-82cb-da81f6bd4343","Type":"ContainerStarted","Data":"720cffc30d9d0f5be617c4f45594d247ee2dfc01e07f74e01830da0dea0126d6"}
Sep 16 13:41:25.525479 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:25.525449    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-44778" event={"ID":"81f3cb95-9aa8-422e-a513-a57d0109d193","Type":"ContainerStarted","Data":"c19d62b313bbaabf8c7c6ea382c87b58f4691641e3b19e29f12a8fdffec1da56"}
Sep 16 13:41:26.532518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:26.532427    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw" event={"ID":"ebfc6e40-8d5b-402f-82cb-da81f6bd4343","Type":"ContainerStarted","Data":"04eb35e7478d1b144f10a21fcef30bea9cf4b7345d367a2941d5c392d4cbc153"}
Sep 16 13:41:26.532926 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:26.532522    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:26.533979 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:26.533953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-44778" event={"ID":"81f3cb95-9aa8-422e-a513-a57d0109d193","Type":"ContainerStarted","Data":"19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39"}
Sep 16 13:41:26.534159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:26.534076    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:26.546926 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:26.546864    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw" podStartSLOduration=1.796553463 podStartE2EDuration="2.5468508s" totalImagesPullingTime="750.297337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:26.54669827 +0000 UTC m=+497.012635197" watchObservedRunningTime="2026-09-16 13:41:26.5468508 +0000 UTC m=+497.012787708"
Sep 16 13:41:26.561844 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:26.561802    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-44778" podStartSLOduration=1.55842931 podStartE2EDuration="2.561789108s" totalImagesPullingTime="1.003359798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:26.559977998 +0000 UTC m=+497.025914916" watchObservedRunningTime="2026-09-16 13:41:26.561789108 +0000 UTC m=+497.027726016"
Sep 16 13:41:37.541234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:37.541205    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:37.541642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:37.541273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-47spw"
Sep 16 13:41:37.597732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:37.597700    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:37.600060 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:37.600030    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-44778" podUID="81f3cb95-9aa8-422e-a513-a57d0109d193" containerName="limitador" containerID="cri-o://19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39" gracePeriod=30
Sep 16 13:41:38.151907 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.151878    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:38.305377 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.305340    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8r9d9\" (UniqueName: \"kubernetes.io/projected/81f3cb95-9aa8-422e-a513-a57d0109d193-kube-api-access-8r9d9\") pod \"81f3cb95-9aa8-422e-a513-a57d0109d193\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") "
Sep 16 13:41:38.305559 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.305388    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f3cb95-9aa8-422e-a513-a57d0109d193-config-file\") pod \"81f3cb95-9aa8-422e-a513-a57d0109d193\" (UID: \"81f3cb95-9aa8-422e-a513-a57d0109d193\") "
Sep 16 13:41:38.305812 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.305786    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f3cb95-9aa8-422e-a513-a57d0109d193-config-file" (OuterVolumeSpecName: "config-file") pod "81f3cb95-9aa8-422e-a513-a57d0109d193" (UID: "81f3cb95-9aa8-422e-a513-a57d0109d193"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:41:38.307799 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.307775    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f3cb95-9aa8-422e-a513-a57d0109d193-kube-api-access-8r9d9" (OuterVolumeSpecName: "kube-api-access-8r9d9") pod "81f3cb95-9aa8-422e-a513-a57d0109d193" (UID: "81f3cb95-9aa8-422e-a513-a57d0109d193"). InnerVolumeSpecName "kube-api-access-8r9d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:41:38.406662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.406624    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r9d9\" (UniqueName: \"kubernetes.io/projected/81f3cb95-9aa8-422e-a513-a57d0109d193-kube-api-access-8r9d9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:41:38.406662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.406653    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/81f3cb95-9aa8-422e-a513-a57d0109d193-config-file\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:41:38.606857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.606770    2556 generic.go:356] "Generic (PLEG): container finished" podID="81f3cb95-9aa8-422e-a513-a57d0109d193" containerID="19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39" exitCode=0
Sep 16 13:41:38.606857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.606818    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-44778"
Sep 16 13:41:38.607293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.606865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-44778" event={"ID":"81f3cb95-9aa8-422e-a513-a57d0109d193","Type":"ContainerDied","Data":"19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39"}
Sep 16 13:41:38.607293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.606927    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-44778" event={"ID":"81f3cb95-9aa8-422e-a513-a57d0109d193","Type":"ContainerDied","Data":"c19d62b313bbaabf8c7c6ea382c87b58f4691641e3b19e29f12a8fdffec1da56"}
Sep 16 13:41:38.607293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.606950    2556 scope.go:117] "RemoveContainer" containerID="19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39"
Sep 16 13:41:38.615962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.615945    2556 scope.go:117] "RemoveContainer" containerID="19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39"
Sep 16 13:41:38.616243 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:41:38.616225    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39\": container with ID starting with 19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39 not found: ID does not exist" containerID="19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39"
Sep 16 13:41:38.616325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.616252    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39"} err="failed to get container status \"19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39\": rpc error: code = NotFound desc = could not find container \"19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39\": container with ID starting with 19c1f57dad68401db7f351d24d69c869ba9f27b6d5611b4ab92686aff3704b39 not found: ID does not exist"
Sep 16 13:41:38.628434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.628408    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:38.630526 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:38.630507    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-44778"]
Sep 16 13:41:40.058521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:40.058491    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f3cb95-9aa8-422e-a513-a57d0109d193" path="/var/lib/kubelet/pods/81f3cb95-9aa8-422e-a513-a57d0109d193/volumes"
Sep 16 13:41:42.859288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:42.859254    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:42.954290 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:42.954254    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:43.321752 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.321722    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"]
Sep 16 13:41:43.322209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.322189    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f3cb95-9aa8-422e-a513-a57d0109d193" containerName="limitador"
Sep 16 13:41:43.322209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.322207    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f3cb95-9aa8-422e-a513-a57d0109d193" containerName="limitador"
Sep 16 13:41:43.322381 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.322279    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f3cb95-9aa8-422e-a513-a57d0109d193" containerName="limitador"
Sep 16 13:41:43.330406 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.330385    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.332545 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.332508    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:41:43.332835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.332817    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 13:41:43.333020 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.332994    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"]
Sep 16 13:41:43.333201 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.333182    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 13:41:43.333349 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.333331    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7hzpz\""
Sep 16 13:41:43.351523 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.351496    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:43.447140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.447080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffxr\" (UniqueName: \"kubernetes.io/projected/82847242-d02e-4187-8ffd-74960e4806d4-kube-api-access-cffxr\") pod \"postgres-69fb4594cf-wk4hb\" (UID: \"82847242-d02e-4187-8ffd-74960e4806d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.447311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.447242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/82847242-d02e-4187-8ffd-74960e4806d4-data\") pod \"postgres-69fb4594cf-wk4hb\" (UID: \"82847242-d02e-4187-8ffd-74960e4806d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.548637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.548603    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffxr\" (UniqueName: \"kubernetes.io/projected/82847242-d02e-4187-8ffd-74960e4806d4-kube-api-access-cffxr\") pod \"postgres-69fb4594cf-wk4hb\" (UID: \"82847242-d02e-4187-8ffd-74960e4806d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.548787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.548682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/82847242-d02e-4187-8ffd-74960e4806d4-data\") pod \"postgres-69fb4594cf-wk4hb\" (UID: \"82847242-d02e-4187-8ffd-74960e4806d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.549120 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.549074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/82847242-d02e-4187-8ffd-74960e4806d4-data\") pod \"postgres-69fb4594cf-wk4hb\" (UID: \"82847242-d02e-4187-8ffd-74960e4806d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.561175 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.561156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffxr\" (UniqueName: \"kubernetes.io/projected/82847242-d02e-4187-8ffd-74960e4806d4-kube-api-access-cffxr\") pod \"postgres-69fb4594cf-wk4hb\" (UID: \"82847242-d02e-4187-8ffd-74960e4806d4\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.641879 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.641785    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:43.913220 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:43.913188    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"]
Sep 16 13:41:43.914652 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:41:43.914618    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82847242_d02e_4187_8ffd_74960e4806d4.slice/crio-5c2310913d1ef3d96f98806636754fb7be38491c240f8b84dd2affeb8c3ee575 WatchSource:0}: Error finding container 5c2310913d1ef3d96f98806636754fb7be38491c240f8b84dd2affeb8c3ee575: Status 404 returned error can't find the container with id 5c2310913d1ef3d96f98806636754fb7be38491c240f8b84dd2affeb8c3ee575
Sep 16 13:41:44.643393 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:44.643357    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb" event={"ID":"82847242-d02e-4187-8ffd-74960e4806d4","Type":"ContainerStarted","Data":"5c2310913d1ef3d96f98806636754fb7be38491c240f8b84dd2affeb8c3ee575"}
Sep 16 13:41:49.679400 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:49.679365    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb" event={"ID":"82847242-d02e-4187-8ffd-74960e4806d4","Type":"ContainerStarted","Data":"e655e511d491562534fee7744c2a03e6400ba29cfbcea4cc8d3063df4b1a0e3f"}
Sep 16 13:41:49.679784 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:49.679453    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:49.695336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:49.695289    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb" podStartSLOduration=1.8560779790000002 podStartE2EDuration="6.695275802s" totalImagesPullingTime="4.839197823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:41:49.693311682 +0000 UTC m=+520.159248587" watchObservedRunningTime="2026-09-16 13:41:49.695275802 +0000 UTC m=+520.161212710"
Sep 16 13:41:55.713445 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:55.713416    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-wk4hb"
Sep 16 13:41:56.020132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:56.020024    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:41:56.625221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:41:56.625187    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:02.728779 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:02.728694    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:02.854365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:02.854329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:02.933511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:02.933475    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:02.990698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:02.990623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:06.673701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:06.673666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:06.772511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:06.772477    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:39.202001 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:39.201957    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:39.292815 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:39.292779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:40.677449 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.677416    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:40.786034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.785994    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:40.816309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.816277    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"]
Sep 16 13:42:40.819879 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.819863    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:40.822491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.822465    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 13:42:40.822491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.822487    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9ljkc\""
Sep 16 13:42:40.822665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.822465    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 13:42:40.830419 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.830397    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"]
Sep 16 13:42:40.848348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.848318    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:40.946548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.946457    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716f3741-ad4a-4d42-9bbc-54b281b53085-openshift-service-ca\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:40.946548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.946497    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-metrics-certs\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:40.946739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.946559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwhjx\" (UniqueName: \"kubernetes.io/projected/716f3741-ad4a-4d42-9bbc-54b281b53085-kube-api-access-pwhjx\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:40.946739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:40.946605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-maas-api-tls\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.047870 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.047841    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwhjx\" (UniqueName: \"kubernetes.io/projected/716f3741-ad4a-4d42-9bbc-54b281b53085-kube-api-access-pwhjx\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.048046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.047875    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-maas-api-tls\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.048046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.047947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716f3741-ad4a-4d42-9bbc-54b281b53085-openshift-service-ca\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.048046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.047968    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-metrics-certs\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.048598 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.048576    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716f3741-ad4a-4d42-9bbc-54b281b53085-openshift-service-ca\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.050478 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.050456    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-maas-api-tls\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.050554 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.050515    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-metrics-certs\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.054982 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.054959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwhjx\" (UniqueName: \"kubernetes.io/projected/716f3741-ad4a-4d42-9bbc-54b281b53085-kube-api-access-pwhjx\") pod \"maas-api-67c45f647d-6nwft\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") " pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.131177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.131146    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:41.181450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.181420    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:41.271962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.271866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:41.309102 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.309053    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"]
Sep 16 13:42:41.312947 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.312929    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.314926 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.314899    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5ph9h\""
Sep 16 13:42:41.315275 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.315255    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 13:42:41.322430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.322406    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"]
Sep 16 13:42:41.331015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.330992    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:42:41.412211 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.412131    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"]
Sep 16 13:42:41.414212 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:42:41.414185    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716f3741_ad4a_4d42_9bbc_54b281b53085.slice/crio-050b86411a4ff0195b35fff91af9bb46d9d98f8cbc554d29f934507962b44f3f WatchSource:0}: Error finding container 050b86411a4ff0195b35fff91af9bb46d9d98f8cbc554d29f934507962b44f3f: Status 404 returned error can't find the container with id 050b86411a4ff0195b35fff91af9bb46d9d98f8cbc554d29f934507962b44f3f
Sep 16 13:42:41.451502 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.451470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed380fe8-2bf2-4ad0-968e-341cbb8a3e17-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x6j2c\" (UID: \"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.451655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.451541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzq9\" (UniqueName: \"kubernetes.io/projected/ed380fe8-2bf2-4ad0-968e-341cbb8a3e17-kube-api-access-qfzq9\") pod \"payload-processing-5b54b7fbb7-x6j2c\" (UID: \"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.552776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.552693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzq9\" (UniqueName: \"kubernetes.io/projected/ed380fe8-2bf2-4ad0-968e-341cbb8a3e17-kube-api-access-qfzq9\") pod \"payload-processing-5b54b7fbb7-x6j2c\" (UID: \"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.552919 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.552787    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed380fe8-2bf2-4ad0-968e-341cbb8a3e17-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x6j2c\" (UID: \"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.553549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.553532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed380fe8-2bf2-4ad0-968e-341cbb8a3e17-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-x6j2c\" (UID: \"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.560619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.560596    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzq9\" (UniqueName: \"kubernetes.io/projected/ed380fe8-2bf2-4ad0-968e-341cbb8a3e17-kube-api-access-qfzq9\") pod \"payload-processing-5b54b7fbb7-x6j2c\" (UID: \"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.625276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.625242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:41.902660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.902634    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"]
Sep 16 13:42:41.904377 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:42:41.904355    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded380fe8_2bf2_4ad0_968e_341cbb8a3e17.slice/crio-00c249cf4f10fafb8c5b5902511ffe0b6bf1691324aacd664ee1f8f597269e75 WatchSource:0}: Error finding container 00c249cf4f10fafb8c5b5902511ffe0b6bf1691324aacd664ee1f8f597269e75: Status 404 returned error can't find the container with id 00c249cf4f10fafb8c5b5902511ffe0b6bf1691324aacd664ee1f8f597269e75
Sep 16 13:42:41.974651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.974611    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c" event={"ID":"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17","Type":"ContainerStarted","Data":"00c249cf4f10fafb8c5b5902511ffe0b6bf1691324aacd664ee1f8f597269e75"}
Sep 16 13:42:41.975896 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:41.975873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" event={"ID":"716f3741-ad4a-4d42-9bbc-54b281b53085","Type":"ContainerStarted","Data":"050b86411a4ff0195b35fff91af9bb46d9d98f8cbc554d29f934507962b44f3f"}
Sep 16 13:42:46.030543 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:46.030506    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c" event={"ID":"ed380fe8-2bf2-4ad0-968e-341cbb8a3e17","Type":"ContainerStarted","Data":"b3aadeb812f0358ebcb8646fc4663ddcf31246e10312fb3b02ee46caa78694cf"}
Sep 16 13:42:46.031055 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:46.030575    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:46.032197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:46.032170    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" event={"ID":"716f3741-ad4a-4d42-9bbc-54b281b53085","Type":"ContainerStarted","Data":"33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05"}
Sep 16 13:42:46.032325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:46.032230    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:46.047322 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:46.047269    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c" podStartSLOduration=1.9174966599999999 podStartE2EDuration="5.047225481s" totalImagesPullingTime="3.129728821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:42:46.046399952 +0000 UTC m=+576.512336914" watchObservedRunningTime="2026-09-16 13:42:46.047225481 +0000 UTC m=+576.513162390"
Sep 16 13:42:46.063104 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:46.063041    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" podStartSLOduration=2.393492649 podStartE2EDuration="6.063028655s" totalImagesPullingTime="3.669536006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:42:46.062367232 +0000 UTC m=+576.528304151" watchObservedRunningTime="2026-09-16 13:42:46.063028655 +0000 UTC m=+576.528965563"
Sep 16 13:42:51.063878 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:51.063797    2556 generic.go:356] "Generic (PLEG): container finished" podID="af20b64c-0972-4684-9fe0-c0f769331b52" containerID="fe3a05b484b64462edd35ff52a39bc5830d523336ff7c738ee0d15da3b2b8bef" exitCode=0
Sep 16 13:42:51.064278 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:51.063873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" event={"ID":"af20b64c-0972-4684-9fe0-c0f769331b52","Type":"ContainerDied","Data":"fe3a05b484b64462edd35ff52a39bc5830d523336ff7c738ee0d15da3b2b8bef"}
Sep 16 13:42:51.064334 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:51.064283    2556 scope.go:117] "RemoveContainer" containerID="fe3a05b484b64462edd35ff52a39bc5830d523336ff7c738ee0d15da3b2b8bef"
Sep 16 13:42:51.065234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:51.065198    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:42:51.065315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:51.065264    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:42:51.065315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:51.065306    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:42:52.044140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:52.044114    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 13:42:52.074034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:52.074005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" event={"ID":"af20b64c-0972-4684-9fe0-c0f769331b52","Type":"ContainerStarted","Data":"06c6759ae2ca3a6f5fa2ce26b85f1780d3fc7e285bc076af5fdf7c1975dbe48c"}
Sep 16 13:42:53.021928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:53.021850    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:42:53.022145 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:53.022059    2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-df6776dd6-gkxhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:42:53.022244 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:53.022144    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" podUID="af20b64c-0972-4684-9fe0-c0f769331b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 16 13:42:53.062436 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:53.062402    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:53.160224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:53.160192    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:42:54.021713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:54.021675    2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-df6776dd6-gkxhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:42:54.021898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:54.021767    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" podUID="af20b64c-0972-4684-9fe0-c0f769331b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 16 13:42:55.021722 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:55.021685    2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-df6776dd6-gkxhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:42:55.022146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:55.021739    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" podUID="af20b64c-0972-4684-9fe0-c0f769331b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 16 13:42:56.021198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:56.021161    2556 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-df6776dd6-gkxhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 16 13:42:56.021386 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:56.021232    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg" podUID="af20b64c-0972-4684-9fe0-c0f769331b52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 16 13:42:57.041309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:57.041269    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-x6j2c"
Sep 16 13:42:57.062737 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:57.062706    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:42:57.063129 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:57.063079    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:42:57.063865 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:42:57.063846    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-df6776dd6-gkxhg"
Sep 16 13:43:00.284552 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:00.284509    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:00.832375 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:00.832339    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:43:14.679748 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:14.679705    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:15.347981 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:15.347941    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:43:24.085250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:24.085213    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:24.937165 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:24.937122    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:43:33.175110 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:33.175002    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:33.236627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:33.236593    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:35.365908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.365877    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"]
Sep 16 13:43:35.369509 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.369492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.371919 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.371901    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xh5w6\""
Sep 16 13:43:35.372454 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.372433    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 13:43:35.372567 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.372468    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 13:43:35.372629 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.372581    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 16 13:43:35.380110 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.380069    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"]
Sep 16 13:43:35.381176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.381157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d30d6c-791a-46e9-90c9-87be00d1269e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.381264 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.381184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.381316 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.381264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqg9\" (UniqueName: \"kubernetes.io/projected/02d30d6c-791a-46e9-90c9-87be00d1269e-kube-api-access-dcqg9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.381354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.381329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.381391 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.381356    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.381391 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.381377    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.384991 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.384971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:35.482010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.481953    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqg9\" (UniqueName: \"kubernetes.io/projected/02d30d6c-791a-46e9-90c9-87be00d1269e-kube-api-access-dcqg9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482120    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d30d6c-791a-46e9-90c9-87be00d1269e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482262    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482532 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482519    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482596 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482576    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.482668 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.482649    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.484666 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.484637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d30d6c-791a-46e9-90c9-87be00d1269e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.484916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.484899    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d30d6c-791a-46e9-90c9-87be00d1269e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.491027 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.490999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqg9\" (UniqueName: \"kubernetes.io/projected/02d30d6c-791a-46e9-90c9-87be00d1269e-kube-api-access-dcqg9\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6btvw\" (UID: \"02d30d6c-791a-46e9-90c9-87be00d1269e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.682117 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.682055    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:35.957420 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.957390    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"]
Sep 16 13:43:35.958692 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:43:35.958665    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d30d6c_791a_46e9_90c9_87be00d1269e.slice/crio-d8fafc218cdf4997f6d915a57cd2a02de4509a0c0e6f23258fb25b2285fbffc8 WatchSource:0}: Error finding container d8fafc218cdf4997f6d915a57cd2a02de4509a0c0e6f23258fb25b2285fbffc8: Status 404 returned error can't find the container with id d8fafc218cdf4997f6d915a57cd2a02de4509a0c0e6f23258fb25b2285fbffc8
Sep 16 13:43:35.960163 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:35.960142    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:43:36.174801 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:36.174769    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:36.338813 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:36.338721    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw" event={"ID":"02d30d6c-791a-46e9-90c9-87be00d1269e","Type":"ContainerStarted","Data":"d8fafc218cdf4997f6d915a57cd2a02de4509a0c0e6f23258fb25b2285fbffc8"}
Sep 16 13:43:36.631934 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:36.631841    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:43:37.348825 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:37.348712    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw" event={"ID":"02d30d6c-791a-46e9-90c9-87be00d1269e","Type":"ContainerStarted","Data":"65c8574f75f0aa98ea00df9c063f083439ff76e291b604376626083f9d00ddd4"}
Sep 16 13:43:37.365466 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:37.365416    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw" podStartSLOduration=1.383220473 podStartE2EDuration="2.365402009s" totalImagesPullingTime="982.181536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:43:37.364186473 +0000 UTC m=+627.830123402" watchObservedRunningTime="2026-09-16 13:43:37.365402009 +0000 UTC m=+627.831338910"
Sep 16 13:43:44.128272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:44.128233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:43:45.582439 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:45.582401    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:43:45.682312 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:45.682280    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:45.682495 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:45.682348    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:45.698954 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:45.698926    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:46.422639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:46.422607    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6btvw"
Sep 16 13:43:59.364969 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:43:59.364931    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:44:01.211424 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:01.211386    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:44:03.646256 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.646215    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"]
Sep 16 13:44:03.649876 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.649851    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.652042 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.652023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 13:44:03.662340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.662318    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"]
Sep 16 13:44:03.738244 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.738218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:44:03.741427 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.741406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.741526 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.741455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.741565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.741532    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.741601 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.741565    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.741601 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.741592    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5c957b-52ab-4eda-97e0-51435c43e3c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.741667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.741608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mwx\" (UniqueName: \"kubernetes.io/projected/1f5c957b-52ab-4eda-97e0-51435c43e3c2-kube-api-access-64mwx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.842751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.842708    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.842751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.842755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.842790    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5c957b-52ab-4eda-97e0-51435c43e3c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.842812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mwx\" (UniqueName: \"kubernetes.io/projected/1f5c957b-52ab-4eda-97e0-51435c43e3c2-kube-api-access-64mwx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.842876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.842951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843258 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.843182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843258 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.843223    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.843372 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.843346    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.845305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.845276    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f5c957b-52ab-4eda-97e0-51435c43e3c2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.845472 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.845457    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f5c957b-52ab-4eda-97e0-51435c43e3c2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.850347 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.850323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mwx\" (UniqueName: \"kubernetes.io/projected/1f5c957b-52ab-4eda-97e0-51435c43e3c2-kube-api-access-64mwx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq\" (UID: \"1f5c957b-52ab-4eda-97e0-51435c43e3c2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:03.958972 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:03.958930    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:04.233996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:04.233968    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"]
Sep 16 13:44:04.235720 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:44:04.235686    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f5c957b_52ab_4eda_97e0_51435c43e3c2.slice/crio-adbb81e555253a6842a0854855489329e09245ddc151415465c6c7fbc53b6399 WatchSource:0}: Error finding container adbb81e555253a6842a0854855489329e09245ddc151415465c6c7fbc53b6399: Status 404 returned error can't find the container with id adbb81e555253a6842a0854855489329e09245ddc151415465c6c7fbc53b6399
Sep 16 13:44:04.508308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:04.508239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq" event={"ID":"1f5c957b-52ab-4eda-97e0-51435c43e3c2","Type":"ContainerStarted","Data":"a950e0fa43fedd9a81e32bd16e3091413d3484cebd293445d126bb33c41358c6"}
Sep 16 13:44:04.508308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:04.508281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq" event={"ID":"1f5c957b-52ab-4eda-97e0-51435c43e3c2","Type":"ContainerStarted","Data":"adbb81e555253a6842a0854855489329e09245ddc151415465c6c7fbc53b6399"}
Sep 16 13:44:04.526030 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:04.525977    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq" podStartSLOduration=1.339538765 podStartE2EDuration="1.525962697s" totalImagesPullingTime="186.423932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:44:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:44:04.523426273 +0000 UTC m=+654.989363183" watchObservedRunningTime="2026-09-16 13:44:04.525962697 +0000 UTC m=+654.991899606"
Sep 16 13:44:06.739236 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:06.739202    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:44:13.959451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:13.959410    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:13.959858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:13.959468    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:13.972505 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:13.972475    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:14.584160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:14.584127    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq"
Sep 16 13:44:17.975465 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:17.975428    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:44:22.980257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:22.980222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:44:23.032943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:44:23.032901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:10.165131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:10.165034    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:10.264749 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:10.264721    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:11.404288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:11.404249    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:11.505688 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:11.505623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:28.989723 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:28.989683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.090503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.090472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.193627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.193592    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.353286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.353191    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.391572 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.391535    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"]
Sep 16 13:45:29.400897 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.400869    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.403367 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.403337    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7d96c7-openshift-default-dockercfg-g8hs7\""
Sep 16 13:45:29.412711 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.412680    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"]
Sep 16 13:45:29.436527 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.436487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.448563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448528    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-certs\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.448563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-data\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.448819 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448651    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgj5r\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-kube-api-access-jgj5r\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.448819 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb835d-2cb2-4c93-af1b-38a95406c14b-istiod-ca-cert\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.448819 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448763    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-podinfo\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.448819 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448793    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-token\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.449019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-socket\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.449019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-credential-socket\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.449019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.448978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-envoy\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.517083 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.517047    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.549948 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.549906    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-podinfo\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.550248 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.550215    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-token\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.550386 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.550274    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-socket\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.550386 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.550315    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-credential-socket\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.550520 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.550417    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-envoy\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.550974 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.550946    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-certs\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551059 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.550999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-credential-socket\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-certs\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551173    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-data\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgj5r\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-kube-api-access-jgj5r\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551324 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb835d-2cb2-4c93-af1b-38a95406c14b-istiod-ca-cert\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551502 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551479    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-data\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551717 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551700    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-socket\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.551960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.551941    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb835d-2cb2-4c93-af1b-38a95406c14b-istiod-ca-cert\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.553228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.553207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-podinfo\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.553749 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.553732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-envoy\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.558773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.558744    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-token\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.559211 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.559194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgj5r\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-kube-api-access-jgj5r\") pod \"e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") " pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.713674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.713638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:29.760682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.760650    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.865301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.865267    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:29.993924 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.993820    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"]
Sep 16 13:45:29.994351 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:45:29.994193    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fb835d_2cb2_4c93_af1b_38a95406c14b.slice/crio-9d9db73697284d94d81a86e0816a15aedfb00ad2ace523a3cb6cbf2e72db52cd WatchSource:0}: Error finding container 9d9db73697284d94d81a86e0816a15aedfb00ad2ace523a3cb6cbf2e72db52cd: Status 404 returned error can't find the container with id 9d9db73697284d94d81a86e0816a15aedfb00ad2ace523a3cb6cbf2e72db52cd
Sep 16 13:45:29.996617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.996585    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:45:29.996708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.996652    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:45:29.996708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:29.996683    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:45:30.020471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:30.020435    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" event={"ID":"29fb835d-2cb2-4c93-af1b-38a95406c14b","Type":"ContainerStarted","Data":"9d9db73697284d94d81a86e0816a15aedfb00ad2ace523a3cb6cbf2e72db52cd"}
Sep 16 13:45:30.238111 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:30.235364    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:30.487506 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:30.487468    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:30.586821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:30.586789    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:30.686459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:30.686422    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:30.975028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:30.975002    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:31.027718 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:31.027678    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" event={"ID":"29fb835d-2cb2-4c93-af1b-38a95406c14b","Type":"ContainerStarted","Data":"bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"}
Sep 16 13:45:31.054278 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:31.054233    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podStartSLOduration=2.054218892 podStartE2EDuration="2.054218892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:31.051562681 +0000 UTC m=+741.517499619" watchObservedRunningTime="2026-09-16 13:45:31.054218892 +0000 UTC m=+741.520155801"
Sep 16 13:45:31.714794 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:31.714762    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:31.719782 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:31.719756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:32.033527 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:32.033440    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:32.034361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:32.034342    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:32.796877 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:32.796815    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:32.857413 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:32.857364    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:45:32.915224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:32.915195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:33.034222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:33.034190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:33.170361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:33.170326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:33.327399 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:33.327370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:34.272269 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.272234    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"]
Sep 16 13:45:34.276287 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.276265    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.278503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.278464    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-fc8c65-serving-cert\""
Sep 16 13:45:34.288723 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.288695    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"]
Sep 16 13:45:34.306887 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.306855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:34.401871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.401840    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4grtk\" (UniqueName: \"kubernetes.io/projected/52c85d93-7a37-4e2b-8143-0a99559cb8e0-kube-api-access-4grtk\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.402053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.401924    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.402053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.401998    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c85d93-7a37-4e2b-8143-0a99559cb8e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.402053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.402024    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-metrics-certs\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.407405 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.407376    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:34.502929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.502893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4grtk\" (UniqueName: \"kubernetes.io/projected/52c85d93-7a37-4e2b-8143-0a99559cb8e0-kube-api-access-4grtk\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.503131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.502951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.503131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.503004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c85d93-7a37-4e2b-8143-0a99559cb8e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.503131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.503026    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-metrics-certs\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.503784 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.503757    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c85d93-7a37-4e2b-8143-0a99559cb8e0-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.505682 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.505658    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-metrics-certs\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.505783 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.505697    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-maas-api-tls\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.508362 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.508339    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:34.510754 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.510735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4grtk\" (UniqueName: \"kubernetes.io/projected/52c85d93-7a37-4e2b-8143-0a99559cb8e0-kube-api-access-4grtk\") pod \"maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.590132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.590036    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:34.865048 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.864971    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"]
Sep 16 13:45:34.867615 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:45:34.867584    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c85d93_7a37_4e2b_8143_0a99559cb8e0.slice/crio-1d160266186e1e160d4c2caec90df18c10663c0d5739fbed59a8e9f47a355b63 WatchSource:0}: Error finding container 1d160266186e1e160d4c2caec90df18c10663c0d5739fbed59a8e9f47a355b63: Status 404 returned error can't find the container with id 1d160266186e1e160d4c2caec90df18c10663c0d5739fbed59a8e9f47a355b63
Sep 16 13:45:34.916539 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:34.916505    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:35.021133 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.021073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:35.054665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.054622    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks" event={"ID":"52c85d93-7a37-4e2b-8143-0a99559cb8e0","Type":"ContainerStarted","Data":"1d160266186e1e160d4c2caec90df18c10663c0d5739fbed59a8e9f47a355b63"}
Sep 16 13:45:35.330198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.330159    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"]
Sep 16 13:45:35.335707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.335673    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.338479 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.338458    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-401a67-serving-cert\""
Sep 16 13:45:35.354127 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.354068    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"]
Sep 16 13:45:35.372332 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.372299    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:35.412614 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.412583    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc010772-b8b9-49ba-9a94-d2570f379621-openshift-service-ca\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.412739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.412644    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-metrics-certs\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.412739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.412729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw42w\" (UniqueName: \"kubernetes.io/projected/cc010772-b8b9-49ba-9a94-d2570f379621-kube-api-access-tw42w\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.412842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.412754    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.444546 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.444503    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:35.513811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.513771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc010772-b8b9-49ba-9a94-d2570f379621-openshift-service-ca\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.514013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.513845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-metrics-certs\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.514013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.513961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw42w\" (UniqueName: \"kubernetes.io/projected/cc010772-b8b9-49ba-9a94-d2570f379621-kube-api-access-tw42w\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.514013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.513996    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.514236 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:45:35.514181    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-serving-cert: secret "maas-api-e2e-inf-a-401a67-serving-cert" not found
Sep 16 13:45:35.514302 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:45:35.514259    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls podName:cc010772-b8b9-49ba-9a94-d2570f379621 nodeName:}" failed. No retries permitted until 2026-09-16 13:45:36.014235829 +0000 UTC m=+746.480172718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls") pod "maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" (UID: "cc010772-b8b9-49ba-9a94-d2570f379621") : secret "maas-api-e2e-inf-a-401a67-serving-cert" not found
Sep 16 13:45:35.514565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.514541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc010772-b8b9-49ba-9a94-d2570f379621-openshift-service-ca\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.516660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.516642    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-metrics-certs\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.522131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.522082    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw42w\" (UniqueName: \"kubernetes.io/projected/cc010772-b8b9-49ba-9a94-d2570f379621-kube-api-access-tw42w\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:35.618468 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.618369    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:35.851837 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.851806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:35.947609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:35.947579    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:36.019589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.019553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:36.022181 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.022150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls\") pod \"maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:36.064139 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.064083    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks" event={"ID":"52c85d93-7a37-4e2b-8143-0a99559cb8e0","Type":"ContainerStarted","Data":"c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113"}
Sep 16 13:45:36.064353 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.064188    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:36.249377 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.249278    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:36.340828 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.340772    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks" podStartSLOduration=1.815906027 podStartE2EDuration="2.340751256s" totalImagesPullingTime="524.845229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:36.080443216 +0000 UTC m=+746.546380144" watchObservedRunningTime="2026-09-16 13:45:36.340751256 +0000 UTC m=+746.806688163"
Sep 16 13:45:36.343625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.343584    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"]
Sep 16 13:45:36.348635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.348612    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.351015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.350843    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7d96c7-serving-cert\""
Sep 16 13:45:36.358976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.358484    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"]
Sep 16 13:45:36.378272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.378241    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:36.423106 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.423055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.423288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.423171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-metrics-certs\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.423288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.423220    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.423369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.423316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mwg7\" (UniqueName: \"kubernetes.io/projected/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-kube-api-access-7mwg7\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.442076 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.442046    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:36.522873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.522844    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"]
Sep 16 13:45:36.524066 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:45:36.524039    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc010772_b8b9_49ba_9a94_d2570f379621.slice/crio-a9a18c0b0d104d3daa1d9a8dbfee65f5cb90db4404f604e6b1e1edccb80d65df WatchSource:0}: Error finding container a9a18c0b0d104d3daa1d9a8dbfee65f5cb90db4404f604e6b1e1edccb80d65df: Status 404 returned error can't find the container with id a9a18c0b0d104d3daa1d9a8dbfee65f5cb90db4404f604e6b1e1edccb80d65df
Sep 16 13:45:36.524252 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.524228    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-metrics-certs\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.524319 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.524286    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.524408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.524393    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mwg7\" (UniqueName: \"kubernetes.io/projected/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-kube-api-access-7mwg7\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.524467 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.524444    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.524609 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:45:36.524595    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-serving-cert: secret "maas-api-e2e-worker-w0-7d96c7-serving-cert" not found
Sep 16 13:45:36.524669 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:45:36.524656    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls podName:ab5c28d4-bc34-47b7-9e3a-14620fd6f16a nodeName:}" failed. No retries permitted until 2026-09-16 13:45:37.024636587 +0000 UTC m=+747.490573480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls") pod "maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" (UID: "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a") : secret "maas-api-e2e-worker-w0-7d96c7-serving-cert" not found
Sep 16 13:45:36.525070 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.525048    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.527032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.527012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-metrics-certs\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.532134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.532083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mwg7\" (UniqueName: \"kubernetes.io/projected/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-kube-api-access-7mwg7\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:36.960236 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:36.960205    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:37.029984 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.029960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:37.032494 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.032472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:37.063878 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.063842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:37.071334 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.071303    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" event={"ID":"cc010772-b8b9-49ba-9a94-d2570f379621","Type":"ContainerStarted","Data":"a9a18c0b0d104d3daa1d9a8dbfee65f5cb90db4404f604e6b1e1edccb80d65df"}
Sep 16 13:45:37.144053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.144015    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:37.266187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.266104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:37.374338 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.374293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:37.546649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.546620    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"]
Sep 16 13:45:37.548269 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:45:37.548246    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5c28d4_bc34_47b7_9e3a_14620fd6f16a.slice/crio-eb1adb92452be4cdaec7e348c93079c8f0365caa7d8718815ba5f5ed8acb255d WatchSource:0}: Error finding container eb1adb92452be4cdaec7e348c93079c8f0365caa7d8718815ba5f5ed8acb255d: Status 404 returned error can't find the container with id eb1adb92452be4cdaec7e348c93079c8f0365caa7d8718815ba5f5ed8acb255d
Sep 16 13:45:37.919697 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:37.919660    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.079350 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.079322    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" event={"ID":"cc010772-b8b9-49ba-9a94-d2570f379621","Type":"ContainerStarted","Data":"af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6"}
Sep 16 13:45:38.079521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.079462    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:38.081016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.080987    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" event={"ID":"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a","Type":"ContainerStarted","Data":"eb1adb92452be4cdaec7e348c93079c8f0365caa7d8718815ba5f5ed8acb255d"}
Sep 16 13:45:38.099255 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.099179    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" podStartSLOduration=2.599751648 podStartE2EDuration="3.099154381s" totalImagesPullingTime="499.402733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:38.097845024 +0000 UTC m=+748.563781935" watchObservedRunningTime="2026-09-16 13:45:38.099154381 +0000 UTC m=+748.565091290"
Sep 16 13:45:38.261646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.261556    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.372319 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.372289    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.485412 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.485377    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.631694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.631594    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:38.719833 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:38.719799    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:39.011476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:39.011443    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:39.092743 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:39.092192    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" event={"ID":"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a","Type":"ContainerStarted","Data":"75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c"}
Sep 16 13:45:39.092963 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:39.092949    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:39.115822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:39.115772    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" podStartSLOduration=2.648162783 podStartE2EDuration="3.115754957s" totalImagesPullingTime="467.592174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:39.113079165 +0000 UTC m=+749.579016089" watchObservedRunningTime="2026-09-16 13:45:39.115754957 +0000 UTC m=+749.581691864"
Sep 16 13:45:39.147081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:39.147049    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:39.294306 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:39.294225    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:40.580235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:40.580172    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:41.270580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:41.270548    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:41.371070 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:41.371018    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:41.779068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:41.779035    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:41.890379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:41.890351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:42.076743 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:42.076665    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:45:42.149811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:42.149780    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:42.242843 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:42.242816    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:42.359387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:42.359301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:42.902242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:42.902213    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:43.593507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:43.593461    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:44.097918 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.097890    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:45:44.247456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.247422    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"]
Sep 16 13:45:44.251333 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.251310    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.254159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.253953    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-d94fd9-serving-cert\""
Sep 16 13:45:44.262651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.262630    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"]
Sep 16 13:45:44.279781 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.279744    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:44.381624 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.381534    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:44.409936 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.409897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-openshift-service-ca\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.410138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.409956    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxmn\" (UniqueName: \"kubernetes.io/projected/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-kube-api-access-tqxmn\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.410138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.410016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-metrics-certs\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.410138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.410062    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-maas-api-tls\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.510986 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.510947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-openshift-service-ca\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.511232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.511000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxmn\" (UniqueName: \"kubernetes.io/projected/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-kube-api-access-tqxmn\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.511232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.511038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-metrics-certs\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.511232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.511081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-maas-api-tls\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.511732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.511695    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-openshift-service-ca\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.513740 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.513710    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-maas-api-tls\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.513866 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.513843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-metrics-certs\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.518663 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.518641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxmn\" (UniqueName: \"kubernetes.io/projected/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-kube-api-access-tqxmn\") pod \"maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.543387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.543352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:44.566933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.566898    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:44.846528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.846503    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"]
Sep 16 13:45:44.848344 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:45:44.848309    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34b0459_cd7b_4393_9b86_b0c8fa3f3753.slice/crio-678ad2c990cfe06ad9d7850453c975d3e994f3e0e9f69bfb10b5c1cbf429c3ab WatchSource:0}: Error finding container 678ad2c990cfe06ad9d7850453c975d3e994f3e0e9f69bfb10b5c1cbf429c3ab: Status 404 returned error can't find the container with id 678ad2c990cfe06ad9d7850453c975d3e994f3e0e9f69bfb10b5c1cbf429c3ab
Sep 16 13:45:44.898293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:44.898264    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.000327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.000295    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.062569 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.062499    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:45:45.131536 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.131459    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55" event={"ID":"a34b0459-cd7b-4393-9b86-b0c8fa3f3753","Type":"ContainerStarted","Data":"678ad2c990cfe06ad9d7850453c975d3e994f3e0e9f69bfb10b5c1cbf429c3ab"}
Sep 16 13:45:45.272967 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.272931    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"]
Sep 16 13:45:45.278742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.278718    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.280828 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.280802    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a2455826-serving-cert\""
Sep 16 13:45:45.289931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.289897    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"]
Sep 16 13:45:45.315603 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.315568    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.413807 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.413770    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:45.419553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.419528    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-maas-api-tls\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.419672 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.419576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-metrics-certs\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.419672 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.419641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9426b30f-33ff-46fd-91f9-055b08cacbd3-openshift-service-ca\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.419749 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.419711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4jkt\" (UniqueName: \"kubernetes.io/projected/9426b30f-33ff-46fd-91f9-055b08cacbd3-kube-api-access-w4jkt\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.520661 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.520622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-metrics-certs\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.520931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.520692    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9426b30f-33ff-46fd-91f9-055b08cacbd3-openshift-service-ca\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.520931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.520733    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4jkt\" (UniqueName: \"kubernetes.io/projected/9426b30f-33ff-46fd-91f9-055b08cacbd3-kube-api-access-w4jkt\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.520931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.520811    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-maas-api-tls\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.522006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.521977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9426b30f-33ff-46fd-91f9-055b08cacbd3-openshift-service-ca\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.523552 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.523528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-metrics-certs\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.523635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.523557    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-maas-api-tls\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.529537 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.529513    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4jkt\" (UniqueName: \"kubernetes.io/projected/9426b30f-33ff-46fd-91f9-055b08cacbd3-kube-api-access-w4jkt\") pod \"maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.597778 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.597740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:45.875265 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.874884    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"]
Sep 16 13:45:45.948133 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:45.948066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:46.045912 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.045877    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:46.111218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.111190    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:45:46.139904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.139867    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt" event={"ID":"9426b30f-33ff-46fd-91f9-055b08cacbd3","Type":"ContainerStarted","Data":"9c7e261d9f45efa2c97050d98a4d08b1ed5be129bf4dd5d4b3c6ef6b35797831"}
Sep 16 13:45:46.141451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.141422    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55" event={"ID":"a34b0459-cd7b-4393-9b86-b0c8fa3f3753","Type":"ContainerStarted","Data":"c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164"}
Sep 16 13:45:46.142130 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.141736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:46.164061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.164003    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55" podStartSLOduration=1.6931881880000001 podStartE2EDuration="2.163982315s" totalImagesPullingTime="470.794127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:46.160385224 +0000 UTC m=+756.626322159" watchObservedRunningTime="2026-09-16 13:45:46.163982315 +0000 UTC m=+756.629919224"
Sep 16 13:45:46.389167 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:46.389129    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:47.152539 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:47.152498    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt" event={"ID":"9426b30f-33ff-46fd-91f9-055b08cacbd3","Type":"ContainerStarted","Data":"5e0454c93f0d89a6ed683a7451914263609a92feca4428a7950677d08b2fc53f"}
Sep 16 13:45:47.152943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:47.152811    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:47.186032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:47.185980    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt" podStartSLOduration=1.8838922139999998 podStartE2EDuration="2.185965988s" totalImagesPullingTime="302.073774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:45:47.184250512 +0000 UTC m=+757.650187420" watchObservedRunningTime="2026-09-16 13:45:47.185965988 +0000 UTC m=+757.651902895"
Sep 16 13:45:47.266710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:47.266675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:47.564501 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:47.564471    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.021928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.021885    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.121963 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.121909    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.223762 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.223731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.324221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.324112    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.541311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.541276    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.648949 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.648919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.728158 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.728124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:48.936333 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:48.936298    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:49.202987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:49.202890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:49.310439 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:49.310406    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:49.413589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:49.413547    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:49.508458 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:49.508365    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:49.609154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:49.609116    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:50.117258 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:50.117218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:50.889209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:50.889171    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:51.537062 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:51.537020    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:52.158903 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:52.158871    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:45:52.240383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:52.240352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:52.342469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:52.342435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:52.694102 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:52.694062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:52.898500 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:52.898403    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:53.034605 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:53.034564    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:53.134835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:53.134801    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:53.167036 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:53.167006    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:45:53.366882 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:53.366843    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:53.459994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:53.459897    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:53.559392 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:53.559356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:54.043002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:54.042966    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:54.148315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:54.148279    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:54.827232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:54.827193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:54.888296 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:54.888253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:54.989795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:54.989730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:55.821564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:55.821485    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:56.224829 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.224796    2556 generic.go:356] "Generic (PLEG): container finished" podID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerID="bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460" exitCode=0
Sep 16 13:45:56.225235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.224869    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" event={"ID":"29fb835d-2cb2-4c93-af1b-38a95406c14b","Type":"ContainerDied","Data":"bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"}
Sep 16 13:45:56.225329 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.225315    2556 scope.go:117] "RemoveContainer" containerID="bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"
Sep 16 13:45:56.226607 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.226575    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:45:56.226679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.226646    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:45:56.226715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.226705    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:45:56.570626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.570544    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:56.672191 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.672162    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:56.998506 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:56.998478    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.098943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.098912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.236545 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.236515    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" event={"ID":"29fb835d-2cb2-4c93-af1b-38a95406c14b","Type":"ContainerStarted","Data":"d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca"}
Sep 16 13:45:57.547284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.547251    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.648138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.648073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:57.714448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.714411    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:45:57.715729 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.715704    2556 patch_prober.go:28] interesting pod/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 13:45:57.715872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:57.715752    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 13:45:58.569801 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:58.569767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:58.715037 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:58.715006    2556 patch_prober.go:28] interesting pod/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 13:45:58.715232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:58.715064    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 13:45:59.484194 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:59.484156    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:45:59.714649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:59.714608    2556 patch_prober.go:28] interesting pod/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 13:45:59.715045 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:45:59.714678    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 13:46:00.307113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.307059    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"]
Sep 16 13:46:00.313177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.313149    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.315673 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.315643    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-6c7792-serving-cert\""
Sep 16 13:46:00.323048 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.323013    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"]
Sep 16 13:46:00.341492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.341374    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:00.403116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.402713    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gb5j\" (UniqueName: \"kubernetes.io/projected/45273968-aaf2-4482-8f10-4c861316753b-kube-api-access-7gb5j\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.403116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.402774    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.403116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.402817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45273968-aaf2-4482-8f10-4c861316753b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.403116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.402861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-metrics-certs\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.439825 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.439787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:00.492734 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.492699    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:00.504409 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.504376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-metrics-certs\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.504795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.504771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb5j\" (UniqueName: \"kubernetes.io/projected/45273968-aaf2-4482-8f10-4c861316753b-kube-api-access-7gb5j\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.504946 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.504930    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.505081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.505066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45273968-aaf2-4482-8f10-4c861316753b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.507210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.507188    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45273968-aaf2-4482-8f10-4c861316753b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.511957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.511934    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-maas-api-tls\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.515140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.515072    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-metrics-certs\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.517879 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.517857    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb5j\" (UniqueName: \"kubernetes.io/projected/45273968-aaf2-4482-8f10-4c861316753b-kube-api-access-7gb5j\") pod \"maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.630993 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.630890    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:00.718110 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.715491    2556 patch_prober.go:28] interesting pod/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 13:46:00.718110 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.715556    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 13:46:00.820644 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.820617    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:00.923419 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.923395    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:00.924296 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:00.924268    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"]
Sep 16 13:46:00.925728 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:46:00.925706    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45273968_aaf2_4482_8f10_4c861316753b.slice/crio-71703a1934a763dd1de49d39ca086d619cc8ce979abc3f6f04e4c666106528af WatchSource:0}: Error finding container 71703a1934a763dd1de49d39ca086d619cc8ce979abc3f6f04e4c666106528af: Status 404 returned error can't find the container with id 71703a1934a763dd1de49d39ca086d619cc8ce979abc3f6f04e4c666106528af
Sep 16 13:46:01.259983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:01.259956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9" event={"ID":"45273968-aaf2-4482-8f10-4c861316753b","Type":"ContainerStarted","Data":"71703a1934a763dd1de49d39ca086d619cc8ce979abc3f6f04e4c666106528af"}
Sep 16 13:46:01.427648 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:01.427613    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:01.714386 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:01.714343    2556 patch_prober.go:28] interesting pod/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 13:46:01.714592 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:01.714414    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 13:46:02.033701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.033611    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:46:02.267637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.267603    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9" event={"ID":"45273968-aaf2-4482-8f10-4c861316753b","Type":"ContainerStarted","Data":"2de7b0443772883bcbf4fe876159af19b60545e2afdf827425b99cac36efeac8"}
Sep 16 13:46:02.267824 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.267734    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:02.284698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.284601    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9" podStartSLOduration=1.977832692 podStartE2EDuration="2.284586489s" totalImagesPullingTime="306.753797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:02.282166015 +0000 UTC m=+772.748102942" watchObservedRunningTime="2026-09-16 13:46:02.284586489 +0000 UTC m=+772.750523388"
Sep 16 13:46:02.452581 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.452550    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:02.714126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.714056    2556 patch_prober.go:28] interesting pod/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 16 13:46:02.714326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:02.714143    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 16 13:46:03.157649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:03.157531    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:03.718646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:03.718616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:46:03.719693 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:03.719670    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:46:03.882176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:03.882142    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:04.568737 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:04.568699    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:05.381183 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:05.381108    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:06.050063 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:06.049995    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:06.874625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:06.874589    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:07.529411 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:07.529381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:08.279145 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:08.279105    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:46:08.347781 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:08.347748    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.141908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.141870    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:46:12.184367 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.184332    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"]
Sep 16 13:46:12.184636 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.184584    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt" podUID="9426b30f-33ff-46fd-91f9-055b08cacbd3" containerName="maas-api" containerID="cri-o://5e0454c93f0d89a6ed683a7451914263609a92feca4428a7950677d08b2fc53f" gracePeriod=30
Sep 16 13:46:12.209960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.209928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.282601 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.282568    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.337444 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.337415    2556 generic.go:356] "Generic (PLEG): container finished" podID="9426b30f-33ff-46fd-91f9-055b08cacbd3" containerID="5e0454c93f0d89a6ed683a7451914263609a92feca4428a7950677d08b2fc53f" exitCode=0
Sep 16 13:46:12.337620 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.337495    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt" event={"ID":"9426b30f-33ff-46fd-91f9-055b08cacbd3","Type":"ContainerDied","Data":"5e0454c93f0d89a6ed683a7451914263609a92feca4428a7950677d08b2fc53f"}
Sep 16 13:46:12.342113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.342069    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.431283 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.430153    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.466661 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.466634    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:46:12.499532 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.499497    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.609583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.609531    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:12.633374 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.633333    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9426b30f-33ff-46fd-91f9-055b08cacbd3-openshift-service-ca\") pod \"9426b30f-33ff-46fd-91f9-055b08cacbd3\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") "
Sep 16 13:46:12.633543 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.633427    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4jkt\" (UniqueName: \"kubernetes.io/projected/9426b30f-33ff-46fd-91f9-055b08cacbd3-kube-api-access-w4jkt\") pod \"9426b30f-33ff-46fd-91f9-055b08cacbd3\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") "
Sep 16 13:46:12.633543 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.633462    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-metrics-certs\") pod \"9426b30f-33ff-46fd-91f9-055b08cacbd3\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") "
Sep 16 13:46:12.633543 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.633500    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-maas-api-tls\") pod \"9426b30f-33ff-46fd-91f9-055b08cacbd3\" (UID: \"9426b30f-33ff-46fd-91f9-055b08cacbd3\") "
Sep 16 13:46:12.633825 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.633796    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9426b30f-33ff-46fd-91f9-055b08cacbd3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9426b30f-33ff-46fd-91f9-055b08cacbd3" (UID: "9426b30f-33ff-46fd-91f9-055b08cacbd3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:46:12.633966 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.633890    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9426b30f-33ff-46fd-91f9-055b08cacbd3-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:12.635879 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.635843    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9426b30f-33ff-46fd-91f9-055b08cacbd3" (UID: "9426b30f-33ff-46fd-91f9-055b08cacbd3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:46:12.635879 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.635861    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9426b30f-33ff-46fd-91f9-055b08cacbd3" (UID: "9426b30f-33ff-46fd-91f9-055b08cacbd3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:46:12.635879 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.635865    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9426b30f-33ff-46fd-91f9-055b08cacbd3-kube-api-access-w4jkt" (OuterVolumeSpecName: "kube-api-access-w4jkt") pod "9426b30f-33ff-46fd-91f9-055b08cacbd3" (UID: "9426b30f-33ff-46fd-91f9-055b08cacbd3"). InnerVolumeSpecName "kube-api-access-w4jkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:46:12.735154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.735059    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4jkt\" (UniqueName: \"kubernetes.io/projected/9426b30f-33ff-46fd-91f9-055b08cacbd3-kube-api-access-w4jkt\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:12.735154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.735119    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:12.735154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.735130    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9426b30f-33ff-46fd-91f9-055b08cacbd3-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:12.917931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:12.917897    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.097856 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.097744    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.196990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.196952    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.336581 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.336549    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.346438 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.346408    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"
Sep 16 13:46:13.346673 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.346438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt" event={"ID":"9426b30f-33ff-46fd-91f9-055b08cacbd3","Type":"ContainerDied","Data":"9c7e261d9f45efa2c97050d98a4d08b1ed5be129bf4dd5d4b3c6ef6b35797831"}
Sep 16 13:46:13.346673 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.346508    2556 scope.go:117] "RemoveContainer" containerID="5e0454c93f0d89a6ed683a7451914263609a92feca4428a7950677d08b2fc53f"
Sep 16 13:46:13.373327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.373293    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"]
Sep 16 13:46:13.375893 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.375871    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2455826-74c854b8b4-vg2wt"]
Sep 16 13:46:13.693966 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.693913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.769219 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.769184    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:13.869565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:13.869516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:14.059481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:14.059398    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9426b30f-33ff-46fd-91f9-055b08cacbd3" path="/var/lib/kubelet/pods/9426b30f-33ff-46fd-91f9-055b08cacbd3/volumes"
Sep 16 13:46:14.771580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:14.771549    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:15.566722 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:15.566687    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:16.445261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:16.445229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:16.632665 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:16.632629    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:16.731603 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:16.731519    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:16.832345 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:16.832306    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:17.370261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:17.370231    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:18.054224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:18.054188    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:18.785181 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:18.785152    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:19.701500 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:19.701466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:20.349766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.349731    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"]
Sep 16 13:46:20.350501 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.350479    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9426b30f-33ff-46fd-91f9-055b08cacbd3" containerName="maas-api"
Sep 16 13:46:20.350582 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.350505    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9426b30f-33ff-46fd-91f9-055b08cacbd3" containerName="maas-api"
Sep 16 13:46:20.350678 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.350665    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9426b30f-33ff-46fd-91f9-055b08cacbd3" containerName="maas-api"
Sep 16 13:46:20.354750 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.354723    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.356619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.356595    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b2821608-serving-cert\""
Sep 16 13:46:20.366138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.364561    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"]
Sep 16 13:46:20.383032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.382997    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:20.411834 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.411802    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.411997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.411850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d20301-566b-4035-bf73-81f06a2577a2-openshift-service-ca\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.411997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.411919    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lslq4\" (UniqueName: \"kubernetes.io/projected/99d20301-566b-4035-bf73-81f06a2577a2-kube-api-access-lslq4\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.411997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.411982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-metrics-certs\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.483760 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.483730    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:20.513351 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.513327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.513511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.513359    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d20301-566b-4035-bf73-81f06a2577a2-openshift-service-ca\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.513511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.513398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lslq4\" (UniqueName: \"kubernetes.io/projected/99d20301-566b-4035-bf73-81f06a2577a2-kube-api-access-lslq4\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.513511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.513440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-metrics-certs\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.513511 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:46:20.513480    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-serving-cert: secret "maas-api-e2e-migrate-b2821608-serving-cert" not found
Sep 16 13:46:20.513720 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:46:20.513549    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls podName:99d20301-566b-4035-bf73-81f06a2577a2 nodeName:}" failed. No retries permitted until 2026-09-16 13:46:21.013530311 +0000 UTC m=+791.479467218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls") pod "maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" (UID: "99d20301-566b-4035-bf73-81f06a2577a2") : secret "maas-api-e2e-migrate-b2821608-serving-cert" not found
Sep 16 13:46:20.514164 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.514142    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d20301-566b-4035-bf73-81f06a2577a2-openshift-service-ca\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.516010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.515991    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-metrics-certs\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.523647 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.523628    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lslq4\" (UniqueName: \"kubernetes.io/projected/99d20301-566b-4035-bf73-81f06a2577a2-kube-api-access-lslq4\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:20.636164 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.636060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:20.830554 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.830519    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:20.929251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:20.929219    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:21.018944 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:21.018913    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:21.021756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:21.021729    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls\") pod \"maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:21.270345 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:21.270256    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:21.370592 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:21.370566    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:21.539394 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:21.539366    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"]
Sep 16 13:46:21.540257 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:46:21.540221    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d20301_566b_4035_bf73_81f06a2577a2.slice/crio-b847022f2e8bbcb07bc2036f50110e2c773c2cc6a33437f006cdd74d1a2e6b06 WatchSource:0}: Error finding container b847022f2e8bbcb07bc2036f50110e2c773c2cc6a33437f006cdd74d1a2e6b06: Status 404 returned error can't find the container with id b847022f2e8bbcb07bc2036f50110e2c773c2cc6a33437f006cdd74d1a2e6b06
Sep 16 13:46:22.171573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:22.171543    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:22.413116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:22.413044    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" event={"ID":"99d20301-566b-4035-bf73-81f06a2577a2","Type":"ContainerStarted","Data":"2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b"}
Sep 16 13:46:22.413311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:22.413171    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" event={"ID":"99d20301-566b-4035-bf73-81f06a2577a2","Type":"ContainerStarted","Data":"b847022f2e8bbcb07bc2036f50110e2c773c2cc6a33437f006cdd74d1a2e6b06"}
Sep 16 13:46:22.413311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:22.413201    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:22.429032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:22.428948    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" podStartSLOduration=2.096300481 podStartE2EDuration="2.428932573s" totalImagesPullingTime="332.632092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:22.428151629 +0000 UTC m=+792.894088560" watchObservedRunningTime="2026-09-16 13:46:22.428932573 +0000 UTC m=+792.894869480"
Sep 16 13:46:22.838585 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:22.838503    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:26.830863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.830824    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"]
Sep 16 13:46:26.833966 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.833940    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.835987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.835963    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"kube-root-ca.crt\""
Sep 16 13:46:26.836522 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.836504    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"default-dockercfg-8j9lj\""
Sep 16 13:46:26.836626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.836535    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"openshift-service-ca.crt\""
Sep 16 13:46:26.836626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.836556    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"simulator-w4-fc8c65-kserve-self-signed-certs\""
Sep 16 13:46:26.842075 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.841780    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"]
Sep 16 13:46:26.877084 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.877047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58rz\" (UniqueName: \"kubernetes.io/projected/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-kube-api-access-x58rz\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.877282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.877125    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-home\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.877282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.877201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-dshm\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.877282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.877236    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tls-certs\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.877282 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.877262    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-model-cache\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.877478 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.877283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tmp-dir\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.949268 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.949231    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:26.978364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-dshm\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978380    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tls-certs\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978415    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-model-cache\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978438    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tmp-dir\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978493    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58rz\" (UniqueName: \"kubernetes.io/projected/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-kube-api-access-x58rz\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978541    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-home\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978845    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-model-cache\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978937 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tmp-dir\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.978973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.978947    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-home\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.980880 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.980858    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-dshm\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.981218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.981199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tls-certs\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:26.986250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:26.986227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58rz\" (UniqueName: \"kubernetes.io/projected/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-kube-api-access-x58rz\") pod \"simulator-w4-fc8c65-kserve-b97895d96-9nwvh\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") " pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:27.144715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:27.144624    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:27.423001 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:27.422976    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"]
Sep 16 13:46:27.424252 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:46:27.424224    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf1b7c57_3f82_40da_8cc7_b8f05dd4d987.slice/crio-bdd5f911c303553ddc2c04760a24b11e76f0647675cad6a12b391a2cf975cea8 WatchSource:0}: Error finding container bdd5f911c303553ddc2c04760a24b11e76f0647675cad6a12b391a2cf975cea8: Status 404 returned error can't find the container with id bdd5f911c303553ddc2c04760a24b11e76f0647675cad6a12b391a2cf975cea8
Sep 16 13:46:27.447325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:27.447293    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh" event={"ID":"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987","Type":"ContainerStarted","Data":"bdd5f911c303553ddc2c04760a24b11e76f0647675cad6a12b391a2cf975cea8"}
Sep 16 13:46:28.424528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:28.424501    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:28.456389 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:28.456348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh" event={"ID":"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987","Type":"ContainerStarted","Data":"d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea"}
Sep 16 13:46:28.474162 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:28.474112    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh" podStartSLOduration=2.474081179 podStartE2EDuration="2.474081179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:46:28.47172146 +0000 UTC m=+798.937658394" watchObservedRunningTime="2026-09-16 13:46:28.474081179 +0000 UTC m=+798.940018078"
Sep 16 13:46:31.532384 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:31.532309    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:46:37.145725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:37.145685    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:37.146119 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:37.145779    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:37.160144 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:37.160113    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:37.523290 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:37.523260    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:46:41.377622 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:41.377588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:42.245141 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:42.245107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:42.649140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:42.649030    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:43.306616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:43.306581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:44.078663 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:44.078629    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:44.773599 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:44.773567    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:45.451357 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:45.451326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:46.270754 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:46.270721    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:47.162838 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:47.162805    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:47.822161 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:47.822125    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:48.583459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:48.583433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:49.277512 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:49.277477    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:50.053110 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:50.053071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:50.726271 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:50.726237    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:51.422431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:51.422391    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:52.178666 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:52.178632    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.062538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.062500    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"]
Sep 16 13:46:58.063032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.062778    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" podUID="99d20301-566b-4035-bf73-81f06a2577a2" containerName="maas-api" containerID="cri-o://2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b" gracePeriod=30
Sep 16 13:46:58.084140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.084082    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.154197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.154154    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.214792 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.214764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.288629 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.288597    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.316422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.316367    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:58.343645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.343620    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.399361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.399331    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d20301-566b-4035-bf73-81f06a2577a2-openshift-service-ca\") pod \"99d20301-566b-4035-bf73-81f06a2577a2\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") "
Sep 16 13:46:58.399540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.399443    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lslq4\" (UniqueName: \"kubernetes.io/projected/99d20301-566b-4035-bf73-81f06a2577a2-kube-api-access-lslq4\") pod \"99d20301-566b-4035-bf73-81f06a2577a2\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") "
Sep 16 13:46:58.399540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.399477    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls\") pod \"99d20301-566b-4035-bf73-81f06a2577a2\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") "
Sep 16 13:46:58.399672 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.399552    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-metrics-certs\") pod \"99d20301-566b-4035-bf73-81f06a2577a2\" (UID: \"99d20301-566b-4035-bf73-81f06a2577a2\") "
Sep 16 13:46:58.399791 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.399759    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d20301-566b-4035-bf73-81f06a2577a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99d20301-566b-4035-bf73-81f06a2577a2" (UID: "99d20301-566b-4035-bf73-81f06a2577a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:46:58.400025 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.400003    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99d20301-566b-4035-bf73-81f06a2577a2-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:58.403131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.403070    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d20301-566b-4035-bf73-81f06a2577a2-kube-api-access-lslq4" (OuterVolumeSpecName: "kube-api-access-lslq4") pod "99d20301-566b-4035-bf73-81f06a2577a2" (UID: "99d20301-566b-4035-bf73-81f06a2577a2"). InnerVolumeSpecName "kube-api-access-lslq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:46:58.405046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.405004    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99d20301-566b-4035-bf73-81f06a2577a2" (UID: "99d20301-566b-4035-bf73-81f06a2577a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:46:58.405194 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.405128    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99d20301-566b-4035-bf73-81f06a2577a2" (UID: "99d20301-566b-4035-bf73-81f06a2577a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:46:58.443716 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.443680    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.501116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.501043    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lslq4\" (UniqueName: \"kubernetes.io/projected/99d20301-566b-4035-bf73-81f06a2577a2-kube-api-access-lslq4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:58.501116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.501078    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:58.501116 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.501123    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99d20301-566b-4035-bf73-81f06a2577a2-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:46:58.645113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.645003    2556 generic.go:356] "Generic (PLEG): container finished" podID="99d20301-566b-4035-bf73-81f06a2577a2" containerID="2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b" exitCode=0
Sep 16 13:46:58.645113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.645055    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"
Sep 16 13:46:58.645294 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.645107    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" event={"ID":"99d20301-566b-4035-bf73-81f06a2577a2","Type":"ContainerDied","Data":"2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b"}
Sep 16 13:46:58.645294 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.645170    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw" event={"ID":"99d20301-566b-4035-bf73-81f06a2577a2","Type":"ContainerDied","Data":"b847022f2e8bbcb07bc2036f50110e2c773c2cc6a33437f006cdd74d1a2e6b06"}
Sep 16 13:46:58.645294 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.645194    2556 scope.go:117] "RemoveContainer" containerID="2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b"
Sep 16 13:46:58.655945 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.655923    2556 scope.go:117] "RemoveContainer" containerID="2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b"
Sep 16 13:46:58.656234 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:46:58.656215    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b\": container with ID starting with 2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b not found: ID does not exist" containerID="2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b"
Sep 16 13:46:58.656297 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.656243    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b"} err="failed to get container status \"2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b\": rpc error: code = NotFound desc = could not find container \"2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b\": container with ID starting with 2d062b378267cc03e6b59fdde00425f0613895fd433b14a66b7c94faf5ebad9b not found: ID does not exist"
Sep 16 13:46:58.667329 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.667304    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"]
Sep 16 13:46:58.669820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.669803    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b2821608-7b47fc68f9-jwmjw"]
Sep 16 13:46:58.693286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.693263    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:58.930572 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:58.930540    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:59.281756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:59.281674    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:59.388732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:59.388699    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:46:59.601425 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:46:59.601331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:00.059896 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:00.059863    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d20301-566b-4035-bf73-81f06a2577a2" path="/var/lib/kubelet/pods/99d20301-566b-4035-bf73-81f06a2577a2/volumes"
Sep 16 13:47:00.406385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:00.406312    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:01.381681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:01.381648    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:01.774032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:01.773997    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:01.869905 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:01.869873    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:02.333076 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:02.333037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:02.976828 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:02.976797    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:03.663120 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.663060    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"]
Sep 16 13:47:03.663685 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.663666    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d20301-566b-4035-bf73-81f06a2577a2" containerName="maas-api"
Sep 16 13:47:03.663781 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.663688    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d20301-566b-4035-bf73-81f06a2577a2" containerName="maas-api"
Sep 16 13:47:03.663842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.663818    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d20301-566b-4035-bf73-81f06a2577a2" containerName="maas-api"
Sep 16 13:47:03.666303 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.666283    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.668380 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.668357    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-74678972-serving-cert\""
Sep 16 13:47:03.676561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.676536    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"]
Sep 16 13:47:03.691736 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.691693    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:03.754589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.754555    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzl7d\" (UniqueName: \"kubernetes.io/projected/c055f744-655d-45a9-a053-878b49d3ea11-kube-api-access-mzl7d\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.754758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.754606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c055f744-655d-45a9-a053-878b49d3ea11-openshift-service-ca\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.754758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.754711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.754758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.754753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-metrics-certs\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.792061 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.792003    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:03.855534 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.855499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzl7d\" (UniqueName: \"kubernetes.io/projected/c055f744-655d-45a9-a053-878b49d3ea11-kube-api-access-mzl7d\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.855721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.855542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c055f744-655d-45a9-a053-878b49d3ea11-openshift-service-ca\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.855721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.855628    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.855721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.855674    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-metrics-certs\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.855894 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:47:03.855815    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-74678972-serving-cert: secret "maas-api-e2e-ait-74678972-serving-cert" not found
Sep 16 13:47:03.855961 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:47:03.855900    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls podName:c055f744-655d-45a9-a053-878b49d3ea11 nodeName:}" failed. No retries permitted until 2026-09-16 13:47:04.355880999 +0000 UTC m=+834.821817885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls") pod "maas-api-e2e-ait-74678972-849b859757-hqtx6" (UID: "c055f744-655d-45a9-a053-878b49d3ea11") : secret "maas-api-e2e-ait-74678972-serving-cert" not found
Sep 16 13:47:03.856403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.856373    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c055f744-655d-45a9-a053-878b49d3ea11-openshift-service-ca\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.858353 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.858326    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-metrics-certs\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:03.864321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:03.864304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzl7d\" (UniqueName: \"kubernetes.io/projected/c055f744-655d-45a9-a053-878b49d3ea11-kube-api-access-mzl7d\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:04.135737 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.135661    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:04.218934 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.218905    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:04.330269 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.330230    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:47:04.361383 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.361337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:04.363906 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.363873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls\") pod \"maas-api-e2e-ait-74678972-849b859757-hqtx6\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:04.581994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.581959    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:04.592891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.592866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:04.856063 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:04.856036    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"]
Sep 16 13:47:04.856622 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:47:04.856600    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc055f744_655d_45a9_a053_878b49d3ea11.slice/crio-02f2a2cd131c38ba948eeb36f60d6a44e812b821b91c45ccffb1b2b51add0ccf WatchSource:0}: Error finding container 02f2a2cd131c38ba948eeb36f60d6a44e812b821b91c45ccffb1b2b51add0ccf: Status 404 returned error can't find the container with id 02f2a2cd131c38ba948eeb36f60d6a44e812b821b91c45ccffb1b2b51add0ccf
Sep 16 13:47:05.299368 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:05.299331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:05.698686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:05.698645    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6" event={"ID":"c055f744-655d-45a9-a053-878b49d3ea11","Type":"ContainerStarted","Data":"411081d2a4561dd051a41c4a3217a0d5ec559a75913258a9930e0f9ef1625dea"}
Sep 16 13:47:05.698875 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:05.698720    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:05.698875 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:05.698745    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6" event={"ID":"c055f744-655d-45a9-a053-878b49d3ea11","Type":"ContainerStarted","Data":"02f2a2cd131c38ba948eeb36f60d6a44e812b821b91c45ccffb1b2b51add0ccf"}
Sep 16 13:47:05.717007 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:05.716961    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6" podStartSLOduration=2.140980196 podStartE2EDuration="2.716944855s" totalImagesPullingTime="575.964659ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:05.71294941 +0000 UTC m=+836.178886361" watchObservedRunningTime="2026-09-16 13:47:05.716944855 +0000 UTC m=+836.182881765"
Sep 16 13:47:05.984765 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:05.984683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:06.791302 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:06.791262    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:11.710672 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:11.710640    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:16.337646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.337595    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"]
Sep 16 13:47:16.342662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.342641    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.345852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.345709    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d96c7\"/\"kube-root-ca.crt\""
Sep 16 13:47:16.345852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.345733    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d96c7\"/\"simulator-w0-7d96c7-kserve-self-signed-certs\""
Sep 16 13:47:16.345852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.345740    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7d96c7\"/\"openshift-service-ca.crt\""
Sep 16 13:47:16.345852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.345799    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d96c7\"/\"default-dockercfg-4k848\""
Sep 16 13:47:16.349518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.349492    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"]
Sep 16 13:47:16.442337 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.442305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:16.482705 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.482675    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85386041-679a-4b3a-a873-6b1299a43674-tls-certs\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.482880 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.482746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-home\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.482880 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.482836    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58swp\" (UniqueName: \"kubernetes.io/projected/85386041-679a-4b3a-a873-6b1299a43674-kube-api-access-58swp\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.482880 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.482868    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-model-cache\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.482999 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.482900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-dshm\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.482999 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.482941    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-tmp-dir\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584024    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-tmp-dir\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584320 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584161    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85386041-679a-4b3a-a873-6b1299a43674-tls-certs\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584320 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-home\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584320 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58swp\" (UniqueName: \"kubernetes.io/projected/85386041-679a-4b3a-a873-6b1299a43674-kube-api-access-58swp\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-model-cache\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-dshm\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584432    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-tmp-dir\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-home\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.584645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.584632    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-model-cache\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.586701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.586670    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-dshm\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.587057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.587036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85386041-679a-4b3a-a873-6b1299a43674-tls-certs\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.592835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.592783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58swp\" (UniqueName: \"kubernetes.io/projected/85386041-679a-4b3a-a873-6b1299a43674-kube-api-access-58swp\") pod \"simulator-w0-7d96c7-kserve-94dbb6f5-vdd67\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") " pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.652829 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.652794    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:16.926216 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:16.926190    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"]
Sep 16 13:47:16.927140 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:47:16.927112    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85386041_679a_4b3a_a873_6b1299a43674.slice/crio-83a31b6a7dcb02a1a82049e2fbcb0e38cfcc0d55934621f507dfd6c32bef3618 WatchSource:0}: Error finding container 83a31b6a7dcb02a1a82049e2fbcb0e38cfcc0d55934621f507dfd6c32bef3618: Status 404 returned error can't find the container with id 83a31b6a7dcb02a1a82049e2fbcb0e38cfcc0d55934621f507dfd6c32bef3618
Sep 16 13:47:17.771936 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:17.771898    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67" event={"ID":"85386041-679a-4b3a-a873-6b1299a43674","Type":"ContainerStarted","Data":"21c3b6585d1fea95e3eb98bc6b7008e706c08ade80aa2a97eb263b6e7cc34e8f"}
Sep 16 13:47:17.772345 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:17.771946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67" event={"ID":"85386041-679a-4b3a-a873-6b1299a43674","Type":"ContainerStarted","Data":"83a31b6a7dcb02a1a82049e2fbcb0e38cfcc0d55934621f507dfd6c32bef3618"}
Sep 16 13:47:17.788646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:17.788593    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67" podStartSLOduration=1.78857247 podStartE2EDuration="1.78857247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:17.786181534 +0000 UTC m=+848.252118462" watchObservedRunningTime="2026-09-16 13:47:17.78857247 +0000 UTC m=+848.254509379"
Sep 16 13:47:22.740363 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:22.740325    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:47:25.566039 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:25.566000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:26.378453 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:26.378397    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:26.653570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:26.653480    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:26.653570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:26.653556    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:26.666026 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:26.666002    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:26.837861 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:26.837827    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:47:27.191741 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:27.191712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:28.039362 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:28.039323    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:28.842203 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:28.842170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:29.554720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:29.554676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:29.673862 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:29.673835    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:30.173938 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:30.173911    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:30.888745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:30.888712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:31.572475 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:31.572439    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:32.421305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:32.421265    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:33.172164 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.172129    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:33.699934 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.699887    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"]
Sep 16 13:47:33.700454 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.700159    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6" podUID="c055f744-655d-45a9-a053-878b49d3ea11" containerName="maas-api" containerID="cri-o://411081d2a4561dd051a41c4a3217a0d5ec559a75913258a9930e0f9ef1625dea" gracePeriod=30
Sep 16 13:47:33.724797 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.724766    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:33.793634 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.793603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:33.855583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.855542    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:33.875430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.875287    2556 generic.go:356] "Generic (PLEG): container finished" podID="c055f744-655d-45a9-a053-878b49d3ea11" containerID="411081d2a4561dd051a41c4a3217a0d5ec559a75913258a9930e0f9ef1625dea" exitCode=0
Sep 16 13:47:33.875430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.875373    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6" event={"ID":"c055f744-655d-45a9-a053-878b49d3ea11","Type":"ContainerDied","Data":"411081d2a4561dd051a41c4a3217a0d5ec559a75913258a9930e0f9ef1625dea"}
Sep 16 13:47:33.931710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.931682    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:33.958321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.958256    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:33.993310 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:33.993280    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:34.061960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.061933    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls\") pod \"c055f744-655d-45a9-a053-878b49d3ea11\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") "
Sep 16 13:47:34.062146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.062009    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c055f744-655d-45a9-a053-878b49d3ea11-openshift-service-ca\") pod \"c055f744-655d-45a9-a053-878b49d3ea11\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") "
Sep 16 13:47:34.062146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.062058    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzl7d\" (UniqueName: \"kubernetes.io/projected/c055f744-655d-45a9-a053-878b49d3ea11-kube-api-access-mzl7d\") pod \"c055f744-655d-45a9-a053-878b49d3ea11\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") "
Sep 16 13:47:34.062254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.062196    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-metrics-certs\") pod \"c055f744-655d-45a9-a053-878b49d3ea11\" (UID: \"c055f744-655d-45a9-a053-878b49d3ea11\") "
Sep 16 13:47:34.062479 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.062451    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c055f744-655d-45a9-a053-878b49d3ea11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c055f744-655d-45a9-a053-878b49d3ea11" (UID: "c055f744-655d-45a9-a053-878b49d3ea11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:47:34.064736 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.064702    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c055f744-655d-45a9-a053-878b49d3ea11" (UID: "c055f744-655d-45a9-a053-878b49d3ea11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:47:34.064883 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.064855    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c055f744-655d-45a9-a053-878b49d3ea11" (UID: "c055f744-655d-45a9-a053-878b49d3ea11"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:47:34.065131 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.065054    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c055f744-655d-45a9-a053-878b49d3ea11-kube-api-access-mzl7d" (OuterVolumeSpecName: "kube-api-access-mzl7d") pod "c055f744-655d-45a9-a053-878b49d3ea11" (UID: "c055f744-655d-45a9-a053-878b49d3ea11"). InnerVolumeSpecName "kube-api-access-mzl7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:47:34.163284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.163246    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzl7d\" (UniqueName: \"kubernetes.io/projected/c055f744-655d-45a9-a053-878b49d3ea11-kube-api-access-mzl7d\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:47:34.163284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.163275    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:47:34.163284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.163284    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c055f744-655d-45a9-a053-878b49d3ea11-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:47:34.163284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.163292    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c055f744-655d-45a9-a053-878b49d3ea11-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:47:34.267738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.267647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:34.883325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.883296    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"
Sep 16 13:47:34.883724 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.883325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6" event={"ID":"c055f744-655d-45a9-a053-878b49d3ea11","Type":"ContainerDied","Data":"02f2a2cd131c38ba948eeb36f60d6a44e812b821b91c45ccffb1b2b51add0ccf"}
Sep 16 13:47:34.883724 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.883381    2556 scope.go:117] "RemoveContainer" containerID="411081d2a4561dd051a41c4a3217a0d5ec559a75913258a9930e0f9ef1625dea"
Sep 16 13:47:34.911853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.911822    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"]
Sep 16 13:47:34.917648 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.917619    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-74678972-849b859757-hqtx6"]
Sep 16 13:47:34.974815 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:34.974779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:35.094284 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:35.094253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:35.533369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:35.533334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:36.060646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:36.060615    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c055f744-655d-45a9-a053-878b49d3ea11" path="/var/lib/kubelet/pods/c055f744-655d-45a9-a053-878b49d3ea11/volumes"
Sep 16 13:47:36.163503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:36.163474    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:36.293307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:36.293278    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:36.694136 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:36.694084    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:37.478249 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:37.478212    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:38.166046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:38.166010    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:38.270215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:38.270183    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:38.370286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:38.370252    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:39.092598 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:39.092562    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:39.958191 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:39.958157    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:40.626273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:40.626233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:41.378174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:41.378133    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:42.058605 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:42.058577    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:42.181830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:42.181802    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:42.381934 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:42.381855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:42.520409 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:42.520373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:42.611671 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:42.611642    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:43.831794 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:43.831757    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:47:51.229146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.229106    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"]
Sep 16 13:47:51.229586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.229572    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c055f744-655d-45a9-a053-878b49d3ea11" containerName="maas-api"
Sep 16 13:47:51.229586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.229587    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c055f744-655d-45a9-a053-878b49d3ea11" containerName="maas-api"
Sep 16 13:47:51.229681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.229672    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c055f744-655d-45a9-a053-878b49d3ea11" containerName="maas-api"
Sep 16 13:47:51.233761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.233744    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.235923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.235869    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-401a67\"/\"kube-root-ca.crt\""
Sep 16 13:47:51.236229 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.236203    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-401a67\"/\"openshift-service-ca.crt\""
Sep 16 13:47:51.236515 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.236496    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-401a67\"/\"test-model-401a67-kserve-self-signed-certs\""
Sep 16 13:47:51.236872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.236853    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-401a67\"/\"default-dockercfg-4rdw8\""
Sep 16 13:47:51.240221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.240200    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"]
Sep 16 13:47:51.333600 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.333566    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:47:51.334774 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.334751    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-dshm\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.334901 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.334823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-model-cache\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.334901 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.334858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-tmp-dir\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.334997 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.334938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5zv\" (UniqueName: \"kubernetes.io/projected/350a81b0-100c-4bde-a717-12a3976db23c-kube-api-access-vz5zv\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.335034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.334995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a81b0-100c-4bde-a717-12a3976db23c-tls-certs\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.335082 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.335034    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-home\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.436677 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.436643    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-dshm\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.436852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.436717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-model-cache\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.436852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.436746    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-tmp-dir\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.436852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.436806    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5zv\" (UniqueName: \"kubernetes.io/projected/350a81b0-100c-4bde-a717-12a3976db23c-kube-api-access-vz5zv\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.436852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.436837    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a81b0-100c-4bde-a717-12a3976db23c-tls-certs\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.437055 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.436877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-home\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.439839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.437635    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-home\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.439839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.437685    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-model-cache\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.439839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.437720    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-tmp-dir\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.440158 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.440136    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-dshm\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.440260 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.440239    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a81b0-100c-4bde-a717-12a3976db23c-tls-certs\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.445822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.445794    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5zv\" (UniqueName: \"kubernetes.io/projected/350a81b0-100c-4bde-a717-12a3976db23c-kube-api-access-vz5zv\") pod \"test-model-401a67-kserve-5d5785656c-68zl9\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") " pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.544503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.544411    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:47:51.815990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.815964    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"]
Sep 16 13:47:51.818293 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:47:51.818265    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod350a81b0_100c_4bde_a717_12a3976db23c.slice/crio-8fc0fadbcf882e706020593339190d6d0d458386832b576c3d43ac683ea6ed66 WatchSource:0}: Error finding container 8fc0fadbcf882e706020593339190d6d0d458386832b576c3d43ac683ea6ed66: Status 404 returned error can't find the container with id 8fc0fadbcf882e706020593339190d6d0d458386832b576c3d43ac683ea6ed66
Sep 16 13:47:51.992809 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.992768    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9" event={"ID":"350a81b0-100c-4bde-a717-12a3976db23c","Type":"ContainerStarted","Data":"ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba"}
Sep 16 13:47:51.992809 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:51.992812    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9" event={"ID":"350a81b0-100c-4bde-a717-12a3976db23c","Type":"ContainerStarted","Data":"8fc0fadbcf882e706020593339190d6d0d458386832b576c3d43ac683ea6ed66"}
Sep 16 13:47:52.010433 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:47:52.010376    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9" podStartSLOduration=1.010359466 podStartE2EDuration="1.010359466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:47:52.007815009 +0000 UTC m=+882.473751951" watchObservedRunningTime="2026-09-16 13:47:52.010359466 +0000 UTC m=+882.476296390"
Sep 16 13:48:01.545385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:01.545299    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:48:01.545385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:01.545353    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:48:01.557851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:01.557824    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:48:02.064369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:02.064334    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:48:02.469376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:02.469340    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:48:02.683444 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:02.683409    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:03.463285 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:03.463224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:04.735818 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:04.735787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:05.505884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:05.505805    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:06.284264 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:06.284234    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:06.973490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:06.973457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:07.678198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:07.678161    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:08.573225 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:08.573190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:09.251707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:09.251677    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:09.925427 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:09.925391    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:11.402733 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:11.402692    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:11.454681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:11.454642    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:12.708084 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:12.708051    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:12.979343 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:12.979255    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:15.559498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:15.559457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.102609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.102559    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"]
Sep 16 13:48:16.102824 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.102794    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55" podUID="a34b0459-cd7b-4393-9b86-b0c8fa3f3753" containerName="maas-api" containerID="cri-o://c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164" gracePeriod=30
Sep 16 13:48:16.131109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.131060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.196397 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.196364    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.254881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.254849    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.322364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.322302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.373580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.373554    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:48:16.375770 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.375745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.384350 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.384317    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-maas-api-tls\") pod \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") "
Sep 16 13:48:16.384440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.384422    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-metrics-certs\") pod \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") "
Sep 16 13:48:16.384478 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.384466    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-openshift-service-ca\") pod \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") "
Sep 16 13:48:16.384541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.384504    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqxmn\" (UniqueName: \"kubernetes.io/projected/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-kube-api-access-tqxmn\") pod \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\" (UID: \"a34b0459-cd7b-4393-9b86-b0c8fa3f3753\") "
Sep 16 13:48:16.384842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.384819    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a34b0459-cd7b-4393-9b86-b0c8fa3f3753" (UID: "a34b0459-cd7b-4393-9b86-b0c8fa3f3753"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:48:16.386613 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.386582    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a34b0459-cd7b-4393-9b86-b0c8fa3f3753" (UID: "a34b0459-cd7b-4393-9b86-b0c8fa3f3753"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:48:16.386710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.386653    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a34b0459-cd7b-4393-9b86-b0c8fa3f3753" (UID: "a34b0459-cd7b-4393-9b86-b0c8fa3f3753"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:48:16.386792 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.386774    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-kube-api-access-tqxmn" (OuterVolumeSpecName: "kube-api-access-tqxmn") pod "a34b0459-cd7b-4393-9b86-b0c8fa3f3753" (UID: "a34b0459-cd7b-4393-9b86-b0c8fa3f3753"). InnerVolumeSpecName "kube-api-access-tqxmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:48:16.485683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.485650    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxmn\" (UniqueName: \"kubernetes.io/projected/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-kube-api-access-tqxmn\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:48:16.485683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.485680    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:48:16.485683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.485690    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:48:16.485919 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.485705    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a34b0459-cd7b-4393-9b86-b0c8fa3f3753-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:48:16.683801 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.683767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:16.782314 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:16.782280    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:17.143865 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.143769    2556 generic.go:356] "Generic (PLEG): container finished" podID="a34b0459-cd7b-4393-9b86-b0c8fa3f3753" containerID="c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164" exitCode=0
Sep 16 13:48:17.143865 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.143822    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"
Sep 16 13:48:17.143865 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.143857    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55" event={"ID":"a34b0459-cd7b-4393-9b86-b0c8fa3f3753","Type":"ContainerDied","Data":"c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164"}
Sep 16 13:48:17.144186 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.143913    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55" event={"ID":"a34b0459-cd7b-4393-9b86-b0c8fa3f3753","Type":"ContainerDied","Data":"678ad2c990cfe06ad9d7850453c975d3e994f3e0e9f69bfb10b5c1cbf429c3ab"}
Sep 16 13:48:17.144186 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.143936    2556 scope.go:117] "RemoveContainer" containerID="c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164"
Sep 16 13:48:17.155304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.155283    2556 scope.go:117] "RemoveContainer" containerID="c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164"
Sep 16 13:48:17.155589 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:48:17.155564    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164\": container with ID starting with c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164 not found: ID does not exist" containerID="c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164"
Sep 16 13:48:17.155702 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.155595    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164"} err="failed to get container status \"c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164\": rpc error: code = NotFound desc = could not find container \"c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164\": container with ID starting with c1ac4a154719313f1574d100d077a49eb713647ece5e55f622747bea57daa164 not found: ID does not exist"
Sep 16 13:48:17.169042 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.169021    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"]
Sep 16 13:48:17.174520 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.174500    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-d94fd9-7d78d7fc76-lhk55"]
Sep 16 13:48:17.573653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:17.573612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:18.059160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:18.059126    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34b0459-cd7b-4393-9b86-b0c8fa3f3753" path="/var/lib/kubelet/pods/a34b0459-cd7b-4393-9b86-b0c8fa3f3753/volumes"
Sep 16 13:48:18.233854 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:18.233813    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:18.894805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:18.894768    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:19.685137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:19.685080    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:20.362354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:20.362313    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:20.920692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:20.920646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.020648 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.020620    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.104615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.104568    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.160137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.160050    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.337355 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.337273    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.549587 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.549557    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.701505 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.701454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:21.950507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:21.950466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:22.045304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:22.045195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:22.146343 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:22.146310    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:22.318455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:22.318370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:23.329395 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:23.329362    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:23.446926 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:23.446889    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:23.818602 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:23.818559    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:24.469172 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:24.469143    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:27.598364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:27.598325    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:28.233382 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:28.233345    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:48:30.393420 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:30.393383    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:31.663235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:31.663204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:31.927701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:31.927664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:32.023452 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:32.023423    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:32.503657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:32.503624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:33.277838 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:33.277810    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:35.152016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:35.151975    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:35.203047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:35.203012    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:35.425645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:35.425616    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:35.513224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:35.513196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:48.871884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:48.871795    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:54.148806 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.148769    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"]
Sep 16 13:48:54.149548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.149525    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34b0459-cd7b-4393-9b86-b0c8fa3f3753" containerName="maas-api"
Sep 16 13:48:54.149622 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.149550    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34b0459-cd7b-4393-9b86-b0c8fa3f3753" containerName="maas-api"
Sep 16 13:48:54.149684 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.149671    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34b0459-cd7b-4393-9b86-b0c8fa3f3753" containerName="maas-api"
Sep 16 13:48:54.153554 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.153527    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.157077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.156405    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6c7792\"/\"openshift-service-ca.crt\""
Sep 16 13:48:54.157077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.156670    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6c7792\"/\"default-dockercfg-pg4w9\""
Sep 16 13:48:54.157077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.156731    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-6c7792\"/\"kube-root-ca.crt\""
Sep 16 13:48:54.157077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.156864    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-6c7792\"/\"test-model-6c7792-kserve-self-signed-certs\""
Sep 16 13:48:54.158985 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.158958    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"]
Sep 16 13:48:54.165326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.165301    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8kx2\" (UniqueName: \"kubernetes.io/projected/020af691-e914-4355-8065-66ec1cc46c46-kube-api-access-m8kx2\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.165459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.165435    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-tmp-dir\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.165528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.165508    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-dshm\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.165586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.165539    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020af691-e914-4355-8065-66ec1cc46c46-tls-certs\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.165586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.165571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-home\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.165694 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.165649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-model-cache\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.171404 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.171370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:54.267192 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-model-cache\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8kx2\" (UniqueName: \"kubernetes.io/projected/020af691-e914-4355-8065-66ec1cc46c46-kube-api-access-m8kx2\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-tmp-dir\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267385    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-dshm\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267414    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020af691-e914-4355-8065-66ec1cc46c46-tls-certs\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267445    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-home\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267622 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-model-cache\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267706 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-tmp-dir\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.267757 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.267748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-home\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.269815 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.269793    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-dshm\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.269972 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.269954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020af691-e914-4355-8065-66ec1cc46c46-tls-certs\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.275032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.275010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8kx2\" (UniqueName: \"kubernetes.io/projected/020af691-e914-4355-8065-66ec1cc46c46-kube-api-access-m8kx2\") pod \"test-model-6c7792-kserve-747d895877-8p7b5\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") " pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.466298 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.466268    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:48:54.749137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:54.749083    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"]
Sep 16 13:48:54.749950 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:48:54.749925    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod020af691_e914_4355_8065_66ec1cc46c46.slice/crio-32fdfedcf3e9a3eacd6caaf2beb69be7d0831201ce6946bcf310ad05e48c9b39 WatchSource:0}: Error finding container 32fdfedcf3e9a3eacd6caaf2beb69be7d0831201ce6946bcf310ad05e48c9b39: Status 404 returned error can't find the container with id 32fdfedcf3e9a3eacd6caaf2beb69be7d0831201ce6946bcf310ad05e48c9b39
Sep 16 13:48:55.380521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:55.380489    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5" event={"ID":"020af691-e914-4355-8065-66ec1cc46c46","Type":"ContainerStarted","Data":"a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0"}
Sep 16 13:48:55.380908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:55.380534    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5" event={"ID":"020af691-e914-4355-8065-66ec1cc46c46","Type":"ContainerStarted","Data":"32fdfedcf3e9a3eacd6caaf2beb69be7d0831201ce6946bcf310ad05e48c9b39"}
Sep 16 13:48:55.397305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:55.397039    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5" podStartSLOduration=1.397008945 podStartE2EDuration="1.397008945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:48:55.397006639 +0000 UTC m=+945.862943550" watchObservedRunningTime="2026-09-16 13:48:55.397008945 +0000 UTC m=+945.862945854"
Sep 16 13:48:56.173728 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:56.173692    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:56.633936 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:56.633837    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:48:57.878760 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:57.878710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:58.686668 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:58.686627    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:48:59.379777 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:48:59.379747    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:00.162892 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:00.162857    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:00.814385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:00.814356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:01.493126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:01.493067    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:02.031045 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:02.031012    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:02.176456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:02.176418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:02.279333 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:02.279305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:02.376642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:02.376546    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:04.466617 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:04.466581    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:49:04.467084 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:04.466629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:49:04.479376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:04.479349    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:49:05.454878 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:05.454848    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:49:18.266873 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:18.266838    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:18.936623 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:18.936586    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:49:24.121932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:24.121893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:24.226720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:24.226676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:24.323220 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:24.323188    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:26.908522 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:26.908479    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:34.854584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:34.854487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:35.616045 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:35.615999    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:41.238228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:41.238194    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:41.350477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:41.350442    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:41.438601 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:41.438557    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:41.579653 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:41.579532    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:42.161202 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:42.161163    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:42.259932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:42.259896    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:42.366009 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:42.365983    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:43.346598 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:43.346560    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:49:44.168747 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:44.168714    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:50.286461 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:50.286423    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:50.386930 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:50.386897    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:50.971578 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:50.971544    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:56.636365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:56.636329    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:56.699970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:56.699930    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:56.873728 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:56.873700    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:57.184424 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:57.184386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:49:57.819395 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:49:57.819361    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:02.600144 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:02.600106    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:02.815495 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:02.815460    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:03.002340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:03.002310    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:03.467325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:03.467286    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:04.224703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:04.224663    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:05.098779 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:05.098744    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:05.859832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:05.859794    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:06.734157 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:06.734128    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:06.836538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:06.836502    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:07.010740 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:07.010649    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:07.296454 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:07.296373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:07.389076 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:07.389035    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:14.245520 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:14.245480    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:50:19.346978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:19.346946    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:29.656808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:29.656775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:30.422842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:30.422809    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:31.184172 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:31.184136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:31.973489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:31.973447    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:32.611990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:32.611957    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:32.893318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:32.893228    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:33.400530 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:33.400499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:33.575290 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:33.575254    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:33.674586 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:33.674561    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:33.998823 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:33.998736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:34.377311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:34.377227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:37.431013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.430970    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"]
Sep 16 13:50:37.434788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.434768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.437308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.437281    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"e2e-conflict-a-cf2b5d9d-kserve-self-signed-certs\""
Sep 16 13:50:37.442249 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.442223    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"]
Sep 16 13:50:37.447990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.447966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxz4\" (UniqueName: \"kubernetes.io/projected/7bdd01d3-418b-4928-91b4-53d02760514a-kube-api-access-8zxz4\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.448146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.448011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-model-cache\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.448212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.448192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdd01d3-418b-4928-91b4-53d02760514a-tls-certs\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.448451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.448337    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-dshm\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.448550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.448447    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-tmp-dir\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.448625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.448520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-home\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.450805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.450783    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:37.549115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdd01d3-418b-4928-91b4-53d02760514a-tls-certs\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549135    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-dshm\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-tmp-dir\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-home\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxz4\" (UniqueName: \"kubernetes.io/projected/7bdd01d3-418b-4928-91b4-53d02760514a-kube-api-access-8zxz4\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549308 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-model-cache\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-model-cache\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549657    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-tmp-dir\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.549767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.549740    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-home\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.551620 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.551604    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-dshm\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.551889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.551872    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdd01d3-418b-4928-91b4-53d02760514a-tls-certs\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.556618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.556597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxz4\" (UniqueName: \"kubernetes.io/projected/7bdd01d3-418b-4928-91b4-53d02760514a-kube-api-access-8zxz4\") pod \"e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:37.746862 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:37.746775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:38.022488 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:38.022455    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"]
Sep 16 13:50:38.023813 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:50:38.023785    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bdd01d3_418b_4928_91b4_53d02760514a.slice/crio-946200a18c784370df2defef2b22cd4f37edb513e5c1b1c4b628827dfbdb3d0e WatchSource:0}: Error finding container 946200a18c784370df2defef2b22cd4f37edb513e5c1b1c4b628827dfbdb3d0e: Status 404 returned error can't find the container with id 946200a18c784370df2defef2b22cd4f37edb513e5c1b1c4b628827dfbdb3d0e
Sep 16 13:50:39.023034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:39.023000    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq" event={"ID":"7bdd01d3-418b-4928-91b4-53d02760514a","Type":"ContainerStarted","Data":"84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847"}
Sep 16 13:50:39.023456 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:39.023049    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq" event={"ID":"7bdd01d3-418b-4928-91b4-53d02760514a","Type":"ContainerStarted","Data":"946200a18c784370df2defef2b22cd4f37edb513e5c1b1c4b628827dfbdb3d0e"}
Sep 16 13:50:39.039051 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:39.039001    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq" podStartSLOduration=2.038985352 podStartE2EDuration="2.038985352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:50:39.038462901 +0000 UTC m=+1049.504399820" watchObservedRunningTime="2026-09-16 13:50:39.038985352 +0000 UTC m=+1049.504922259"
Sep 16 13:50:39.636014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:39.635980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:39.735645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:39.735607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:40.995546 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:40.995489    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:41.676423 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:41.676382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:42.578128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:42.578076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:43.273751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:43.273716    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:43.927725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:43.927695    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:44.688141 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:44.688107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:45.248905 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:45.248873    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:50:45.365280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:45.365233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:47.747970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:47.747927    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:47.748393 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:47.747993    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:47.760751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:47.760724    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:48.091535 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:48.091454    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:50:50.625057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:50.625019    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:50.724519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:50.724488    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:50.812798 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:50.812761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:50.864316 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:50.864287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:51.197821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:51.197789    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:51.299923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:51.299883    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:52.533210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:52.533178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:52.882354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:52.882261    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:53.570286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:53.570247    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:53.685770 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:53.685729    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:54.239454 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:54.239421    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:54.912309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:54.912278    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:55.679892 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:55.679823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:56.360514 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:56.360483    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:57.069961 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:57.069929    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:57.293943 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:57.293909    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:57.394955 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:57.394870    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:57.554595 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:57.554569    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:57.770696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:57.770663    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:57.923010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:57.922974    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:58.024769 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:58.024683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:58.124028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:58.123994    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:58.687949 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:58.687913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:50:59.393070 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:50:59.393008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:00.083486 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:00.083452    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:00.850800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:00.850763    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:02.572159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:02.572047    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:04.013276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:04.013242    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:04.064733 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:04.064700    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:04.141154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:04.141119    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:04.243746 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:04.243708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:04.354388 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:04.354312    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:05.404736 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:05.404704    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:05.505451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:05.505408    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:05.981270 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:05.981239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:06.059151 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:06.059111    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:06.159396 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:06.159368    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:06.227166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:06.227139    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:06.529898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:06.529866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:06.647443 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:06.647407    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:07.432075 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:07.432044    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:07.725017 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:07.724923    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"]
Sep 16 13:51:07.746341 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:07.746315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:08.111461 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:08.111373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:08.822235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:08.822204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:09.489215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:09.489145    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:10.248401 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:10.248366    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:10.348784 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:10.348747    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:11.157126 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:11.157067    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:11.475290 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:11.475258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:11.675353 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:11.675320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:11.775964 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:11.775861    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:14.674226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:14.674188    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:14.927499 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:14.927397    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:15.026304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:15.026269    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:15.593908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:15.593878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:16.380593 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:16.380556    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:17.276014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:17.275984    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:17.955221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:17.955191    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:18.056077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:18.056027    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:18.156698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:18.156645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:18.406356 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:18.406278    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:18.501946 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:18.501913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:21.937414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:21.937379    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:51:22.745239 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:22.745199    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9" podUID="350a81b0-100c-4bde-a717-12a3976db23c" containerName="main" containerID="cri-o://ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba" gracePeriod=45
Sep 16 13:51:23.014121 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.014075    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:51:23.135847 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.135803    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a81b0-100c-4bde-a717-12a3976db23c-tls-certs\") pod \"350a81b0-100c-4bde-a717-12a3976db23c\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") "
Sep 16 13:51:23.136056 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.135865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-tmp-dir\") pod \"350a81b0-100c-4bde-a717-12a3976db23c\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") "
Sep 16 13:51:23.136056 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.135901    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz5zv\" (UniqueName: \"kubernetes.io/projected/350a81b0-100c-4bde-a717-12a3976db23c-kube-api-access-vz5zv\") pod \"350a81b0-100c-4bde-a717-12a3976db23c\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") "
Sep 16 13:51:23.136056 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136008    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-home\") pod \"350a81b0-100c-4bde-a717-12a3976db23c\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") "
Sep 16 13:51:23.136056 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136035    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-dshm\") pod \"350a81b0-100c-4bde-a717-12a3976db23c\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") "
Sep 16 13:51:23.136326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136062    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-model-cache\") pod \"350a81b0-100c-4bde-a717-12a3976db23c\" (UID: \"350a81b0-100c-4bde-a717-12a3976db23c\") "
Sep 16 13:51:23.136326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136273    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "350a81b0-100c-4bde-a717-12a3976db23c" (UID: "350a81b0-100c-4bde-a717-12a3976db23c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:23.136551 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136526    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-model-cache" (OuterVolumeSpecName: "model-cache") pod "350a81b0-100c-4bde-a717-12a3976db23c" (UID: "350a81b0-100c-4bde-a717-12a3976db23c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:23.136551 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136538    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-home" (OuterVolumeSpecName: "home") pod "350a81b0-100c-4bde-a717-12a3976db23c" (UID: "350a81b0-100c-4bde-a717-12a3976db23c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:23.136835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.136545    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:23.138451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.138420    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/350a81b0-100c-4bde-a717-12a3976db23c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "350a81b0-100c-4bde-a717-12a3976db23c" (UID: "350a81b0-100c-4bde-a717-12a3976db23c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:51:23.138570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.138469    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350a81b0-100c-4bde-a717-12a3976db23c-kube-api-access-vz5zv" (OuterVolumeSpecName: "kube-api-access-vz5zv") pod "350a81b0-100c-4bde-a717-12a3976db23c" (UID: "350a81b0-100c-4bde-a717-12a3976db23c"). InnerVolumeSpecName "kube-api-access-vz5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:51:23.138570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.138535    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-dshm" (OuterVolumeSpecName: "dshm") pod "350a81b0-100c-4bde-a717-12a3976db23c" (UID: "350a81b0-100c-4bde-a717-12a3976db23c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:23.237996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.237955    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5zv\" (UniqueName: \"kubernetes.io/projected/350a81b0-100c-4bde-a717-12a3976db23c-kube-api-access-vz5zv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:23.237996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.237996    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:23.238226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.238011    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:23.238226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.238024    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/350a81b0-100c-4bde-a717-12a3976db23c-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:23.238226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.238037    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/350a81b0-100c-4bde-a717-12a3976db23c-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:23.310432 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.310334    2556 generic.go:356] "Generic (PLEG): container finished" podID="350a81b0-100c-4bde-a717-12a3976db23c" containerID="ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba" exitCode=0
Sep 16 13:51:23.310432 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.310401    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"
Sep 16 13:51:23.310432 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.310419    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9" event={"ID":"350a81b0-100c-4bde-a717-12a3976db23c","Type":"ContainerDied","Data":"ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba"}
Sep 16 13:51:23.310717 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.310464    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9" event={"ID":"350a81b0-100c-4bde-a717-12a3976db23c","Type":"ContainerDied","Data":"8fc0fadbcf882e706020593339190d6d0d458386832b576c3d43ac683ea6ed66"}
Sep 16 13:51:23.310717 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.310481    2556 scope.go:117] "RemoveContainer" containerID="ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba"
Sep 16 13:51:23.322379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.322361    2556 scope.go:117] "RemoveContainer" containerID="ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba"
Sep 16 13:51:23.322707 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:51:23.322685    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba\": container with ID starting with ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba not found: ID does not exist" containerID="ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba"
Sep 16 13:51:23.322782 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.322718    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba"} err="failed to get container status \"ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba\": rpc error: code = NotFound desc = could not find container \"ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba\": container with ID starting with ca9877b3c995808bdc1b30fa81cd51b164132e9f1b63136e3d16643d9bb88eba not found: ID does not exist"
Sep 16 13:51:23.333177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.333152    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"]
Sep 16 13:51:23.341448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.341427    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-401a67/test-model-401a67-kserve-5d5785656c-68zl9"]
Sep 16 13:51:23.480743 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:23.480711    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:24.060527 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:24.060494    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="350a81b0-100c-4bde-a717-12a3976db23c" path="/var/lib/kubelet/pods/350a81b0-100c-4bde-a717-12a3976db23c/volumes"
Sep 16 13:51:35.848701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:35.848666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:36.469821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:36.469775    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"]
Sep 16 13:51:36.488162 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:36.488127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:36.716214 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:36.716183    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:37.389998 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:37.389963    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:38.183202 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:38.183169    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:38.871630 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:38.871600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:39.659548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:39.659506    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:39.897862 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:39.897821    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:40.258297 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.258256    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:40.423360 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.423327    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:40.537462 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.537376    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"]
Sep 16 13:51:40.537890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.537876    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="350a81b0-100c-4bde-a717-12a3976db23c" containerName="main"
Sep 16 13:51:40.537969 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.537892    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="350a81b0-100c-4bde-a717-12a3976db23c" containerName="main"
Sep 16 13:51:40.538027 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.538022    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="350a81b0-100c-4bde-a717-12a3976db23c" containerName="main"
Sep 16 13:51:40.541349 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.541329    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.543573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.543552    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-284b60-openshift-default-dockercfg-lcsg6\""
Sep 16 13:51:40.554295 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.554262    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"]
Sep 16 13:51:40.573987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.573940    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:40.716128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.716080    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:40.718365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-envoy\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718392    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-podinfo\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718439    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-credential-socket\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmjw\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-kube-api-access-hsmjw\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718555    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-data\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-certs\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-token\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718620    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a453e-4175-4b96-8d9e-9ef0da349fe9-istiod-ca-cert\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.718801 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.718702    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-socket\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.819910    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-envoy\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.819990    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-podinfo\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-credential-socket\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmjw\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-kube-api-access-hsmjw\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820139    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-data\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-certs\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820550    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-data\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820580 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820573    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-token\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820942 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820478    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-credential-socket\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820942 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-certs\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820942 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820643    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a453e-4175-4b96-8d9e-9ef0da349fe9-istiod-ca-cert\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.820942 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-socket\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.821197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.820977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-socket\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.821262 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.821223    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a453e-4175-4b96-8d9e-9ef0da349fe9-istiod-ca-cert\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.822457 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.822435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-envoy\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.822764 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.822748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-podinfo\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.827498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.827463    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-token\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.827608 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.827571    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmjw\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-kube-api-access-hsmjw\") pod \"e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") " pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:40.855474 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:40.855425    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:41.131727 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.131696    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"]
Sep 16 13:51:41.133019 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:51:41.132994    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052a453e_4175_4b96_8d9e_9ef0da349fe9.slice/crio-212e1f4f7c49262d2689b8f2fddd391ba93e30c3758be49bc3e6fa6dc24fa33d WatchSource:0}: Error finding container 212e1f4f7c49262d2689b8f2fddd391ba93e30c3758be49bc3e6fa6dc24fa33d: Status 404 returned error can't find the container with id 212e1f4f7c49262d2689b8f2fddd391ba93e30c3758be49bc3e6fa6dc24fa33d
Sep 16 13:51:41.136209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.136173    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:51:41.136315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.136246    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:51:41.136315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.136274    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:51:41.196909 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.196866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:41.424081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.424050    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" event={"ID":"052a453e-4175-4b96-8d9e-9ef0da349fe9","Type":"ContainerStarted","Data":"164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"}
Sep 16 13:51:41.424273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.424121    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" event={"ID":"052a453e-4175-4b96-8d9e-9ef0da349fe9","Type":"ContainerStarted","Data":"212e1f4f7c49262d2689b8f2fddd391ba93e30c3758be49bc3e6fa6dc24fa33d"}
Sep 16 13:51:41.443047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.443003    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podStartSLOduration=1.442987459 podStartE2EDuration="1.442987459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:41.441546727 +0000 UTC m=+1111.907483669" watchObservedRunningTime="2026-09-16 13:51:41.442987459 +0000 UTC m=+1111.908924367"
Sep 16 13:51:41.855803 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.855706    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:41.985713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:41.985675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:42.861506 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:42.861477    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:43.074470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:43.074433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:43.423536 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:43.423495    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:43.437214 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:43.437190    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:43.438206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:43.438185    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:51:43.512577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:43.512546    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:43.923158 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:43.923080    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:44.241193 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:44.241109    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:51:44.624969 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:44.624878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:45.600226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:45.600196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:45.701228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:45.701193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:46.317141 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.317109    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:46.374422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.374388    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:46.436626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.436579    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"]
Sep 16 13:51:46.436836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.436802    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" podUID="cc010772-b8b9-49ba-9a94-d2570f379621" containerName="maas-api" containerID="cri-o://af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6" gracePeriod=30
Sep 16 13:51:46.465073 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.464376    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:46.528708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.528679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:46.594075 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.593993    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:46.692259 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.692234    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:51:46.782445 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.782405    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-metrics-certs\") pod \"cc010772-b8b9-49ba-9a94-d2570f379621\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") "
Sep 16 13:51:46.782657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.782465    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw42w\" (UniqueName: \"kubernetes.io/projected/cc010772-b8b9-49ba-9a94-d2570f379621-kube-api-access-tw42w\") pod \"cc010772-b8b9-49ba-9a94-d2570f379621\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") "
Sep 16 13:51:46.782657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.782514    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc010772-b8b9-49ba-9a94-d2570f379621-openshift-service-ca\") pod \"cc010772-b8b9-49ba-9a94-d2570f379621\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") "
Sep 16 13:51:46.782657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.782575    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls\") pod \"cc010772-b8b9-49ba-9a94-d2570f379621\" (UID: \"cc010772-b8b9-49ba-9a94-d2570f379621\") "
Sep 16 13:51:46.782929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.782905    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc010772-b8b9-49ba-9a94-d2570f379621-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc010772-b8b9-49ba-9a94-d2570f379621" (UID: "cc010772-b8b9-49ba-9a94-d2570f379621"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:51:46.784784 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.784757    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cc010772-b8b9-49ba-9a94-d2570f379621" (UID: "cc010772-b8b9-49ba-9a94-d2570f379621"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:51:46.784886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.784827    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc010772-b8b9-49ba-9a94-d2570f379621" (UID: "cc010772-b8b9-49ba-9a94-d2570f379621"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:51:46.784886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.784827    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc010772-b8b9-49ba-9a94-d2570f379621-kube-api-access-tw42w" (OuterVolumeSpecName: "kube-api-access-tw42w") pod "cc010772-b8b9-49ba-9a94-d2570f379621" (UID: "cc010772-b8b9-49ba-9a94-d2570f379621"). InnerVolumeSpecName "kube-api-access-tw42w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:51:46.883983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.883893    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw42w\" (UniqueName: \"kubernetes.io/projected/cc010772-b8b9-49ba-9a94-d2570f379621-kube-api-access-tw42w\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:46.883983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.883921    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc010772-b8b9-49ba-9a94-d2570f379621-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:46.883983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.883934    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:46.883983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:46.883944    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc010772-b8b9-49ba-9a94-d2570f379621-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:47.097674 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.097646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:47.197760 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.197728    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:47.468507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.468410    2556 generic.go:356] "Generic (PLEG): container finished" podID="cc010772-b8b9-49ba-9a94-d2570f379621" containerID="af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6" exitCode=0
Sep 16 13:51:47.468507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.468447    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"
Sep 16 13:51:47.468507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.468489    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" event={"ID":"cc010772-b8b9-49ba-9a94-d2570f379621","Type":"ContainerDied","Data":"af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6"}
Sep 16 13:51:47.468758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.468538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp" event={"ID":"cc010772-b8b9-49ba-9a94-d2570f379621","Type":"ContainerDied","Data":"a9a18c0b0d104d3daa1d9a8dbfee65f5cb90db4404f604e6b1e1edccb80d65df"}
Sep 16 13:51:47.468758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.468560    2556 scope.go:117] "RemoveContainer" containerID="af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6"
Sep 16 13:51:47.479598 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.479577    2556 scope.go:117] "RemoveContainer" containerID="af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6"
Sep 16 13:51:47.479921 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:51:47.479903    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6\": container with ID starting with af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6 not found: ID does not exist" containerID="af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6"
Sep 16 13:51:47.479990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.479936    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6"} err="failed to get container status \"af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6\": rpc error: code = NotFound desc = could not find container \"af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6\": container with ID starting with af61fd45f3d79bb70f550533d216fbb9e6d18355edfeb1ddf9a00bc14196c9e6 not found: ID does not exist"
Sep 16 13:51:47.493209 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.493161    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"]
Sep 16 13:51:47.498626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.498589    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-401a67-765c7df5ff-c5zsp"]
Sep 16 13:51:47.530400 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:47.530370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:48.065635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.065600    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc010772-b8b9-49ba-9a94-d2570f379621" path="/var/lib/kubelet/pods/cc010772-b8b9-49ba-9a94-d2570f379621/volumes"
Sep 16 13:51:48.300255 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.300221    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:48.432257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.432220    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"]
Sep 16 13:51:48.432752 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.432735    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc010772-b8b9-49ba-9a94-d2570f379621" containerName="maas-api"
Sep 16 13:51:48.432802 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.432755    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc010772-b8b9-49ba-9a94-d2570f379621" containerName="maas-api"
Sep 16 13:51:48.432860 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.432851    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc010772-b8b9-49ba-9a94-d2570f379621" containerName="maas-api"
Sep 16 13:51:48.437336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.437318    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.440113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.440065    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"e2e-conflict-b-cf2b5d9d-kserve-self-signed-certs\""
Sep 16 13:51:48.444147 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.444123    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"]
Sep 16 13:51:48.500132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.500081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-home\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.500276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.500146    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-tmp-dir\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.500276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.500225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-dshm\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.500276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.500266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-model-cache\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.500448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.500295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5c8c5a-8da8-44de-887c-5dd8af358572-tls-certs\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.500448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.500414    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqz8k\" (UniqueName: \"kubernetes.io/projected/cf5c8c5a-8da8-44de-887c-5dd8af358572-kube-api-access-kqz8k\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.600996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.600959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-model-cache\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601182 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5c8c5a-8da8-44de-887c-5dd8af358572-tls-certs\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601182 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601052    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqz8k\" (UniqueName: \"kubernetes.io/projected/cf5c8c5a-8da8-44de-887c-5dd8af358572-kube-api-access-kqz8k\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601182 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-home\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601182 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-tmp-dir\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-dshm\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601484 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-model-cache\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601484 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-home\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.601575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.601557    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-tmp-dir\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.603683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.603645    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-dshm\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.603827 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.603809    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5c8c5a-8da8-44de-887c-5dd8af358572-tls-certs\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.608155 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.608135    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqz8k\" (UniqueName: \"kubernetes.io/projected/cf5c8c5a-8da8-44de-887c-5dd8af358572-kube-api-access-kqz8k\") pod \"e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") " pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.739402 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.739314    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:48.748843 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.748811    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:48.839561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.839532    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:48.983222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:48.983186    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:49.020837 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.020813    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"]
Sep 16 13:51:49.022580 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:51:49.022555    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf5c8c5a_8da8_44de_887c_5dd8af358572.slice/crio-3a9966d8b5b8346c0dd571c9c47c0065ccddcc86dcf28ac4e253d25c31cdf0e0 WatchSource:0}: Error finding container 3a9966d8b5b8346c0dd571c9c47c0065ccddcc86dcf28ac4e253d25c31cdf0e0: Status 404 returned error can't find the container with id 3a9966d8b5b8346c0dd571c9c47c0065ccddcc86dcf28ac4e253d25c31cdf0e0
Sep 16 13:51:49.100631 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.100603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:49.412452 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.412364    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:49.484606 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.484569    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv" event={"ID":"cf5c8c5a-8da8-44de-887c-5dd8af358572","Type":"ContainerStarted","Data":"2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4"}
Sep 16 13:51:49.484784 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.484619    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv" event={"ID":"cf5c8c5a-8da8-44de-887c-5dd8af358572","Type":"ContainerStarted","Data":"3a9966d8b5b8346c0dd571c9c47c0065ccddcc86dcf28ac4e253d25c31cdf0e0"}
Sep 16 13:51:49.502481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.502429    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv" podStartSLOduration=1.502412115 podStartE2EDuration="1.502412115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:51:49.500438857 +0000 UTC m=+1119.966375766" watchObservedRunningTime="2026-09-16 13:51:49.502412115 +0000 UTC m=+1119.968349022"
Sep 16 13:51:49.797960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.797913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:49.972909 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:49.972876    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:50.152939 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:50.152855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:50.675912 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:50.675878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:51.448414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.448382    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:51.491450 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.491402    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5" podUID="020af691-e914-4355-8065-66ec1cc46c46" containerName="main" containerID="cri-o://a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0" gracePeriod=45
Sep 16 13:51:51.591766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.591729    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:51.690745 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.690684    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:51.755492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.755459    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:51:51.795299 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.795270    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:51.839178 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839140    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020af691-e914-4355-8065-66ec1cc46c46-tls-certs\") pod \"020af691-e914-4355-8065-66ec1cc46c46\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") "
Sep 16 13:51:51.839378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839187    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-home\") pod \"020af691-e914-4355-8065-66ec1cc46c46\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") "
Sep 16 13:51:51.839378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839211    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-dshm\") pod \"020af691-e914-4355-8065-66ec1cc46c46\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") "
Sep 16 13:51:51.839378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839246    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8kx2\" (UniqueName: \"kubernetes.io/projected/020af691-e914-4355-8065-66ec1cc46c46-kube-api-access-m8kx2\") pod \"020af691-e914-4355-8065-66ec1cc46c46\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") "
Sep 16 13:51:51.839378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839285    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-model-cache\") pod \"020af691-e914-4355-8065-66ec1cc46c46\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") "
Sep 16 13:51:51.839378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839315    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-tmp-dir\") pod \"020af691-e914-4355-8065-66ec1cc46c46\" (UID: \"020af691-e914-4355-8065-66ec1cc46c46\") "
Sep 16 13:51:51.839671 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839533    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-model-cache" (OuterVolumeSpecName: "model-cache") pod "020af691-e914-4355-8065-66ec1cc46c46" (UID: "020af691-e914-4355-8065-66ec1cc46c46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:51.839671 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839571    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-home" (OuterVolumeSpecName: "home") pod "020af691-e914-4355-8065-66ec1cc46c46" (UID: "020af691-e914-4355-8065-66ec1cc46c46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:51.839671 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839641    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "020af691-e914-4355-8065-66ec1cc46c46" (UID: "020af691-e914-4355-8065-66ec1cc46c46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:51.839810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839767    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:51.839810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839786    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:51.839810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.839800    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:51.841785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.841763    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/020af691-e914-4355-8065-66ec1cc46c46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "020af691-e914-4355-8065-66ec1cc46c46" (UID: "020af691-e914-4355-8065-66ec1cc46c46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:51:51.841785 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.841775    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-dshm" (OuterVolumeSpecName: "dshm") pod "020af691-e914-4355-8065-66ec1cc46c46" (UID: "020af691-e914-4355-8065-66ec1cc46c46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:51:51.841927 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.841780    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020af691-e914-4355-8065-66ec1cc46c46-kube-api-access-m8kx2" (OuterVolumeSpecName: "kube-api-access-m8kx2") pod "020af691-e914-4355-8065-66ec1cc46c46" (UID: "020af691-e914-4355-8065-66ec1cc46c46"). InnerVolumeSpecName "kube-api-access-m8kx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:51:51.940806 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.940766    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/020af691-e914-4355-8065-66ec1cc46c46-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:51.940806 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.940798    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/020af691-e914-4355-8065-66ec1cc46c46-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:51.940806 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:51.940811    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8kx2\" (UniqueName: \"kubernetes.io/projected/020af691-e914-4355-8065-66ec1cc46c46-kube-api-access-m8kx2\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:51:52.509357 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.509293    2556 generic.go:356] "Generic (PLEG): container finished" podID="020af691-e914-4355-8065-66ec1cc46c46" containerID="a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0" exitCode=0
Sep 16 13:51:52.510058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.509463    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5" event={"ID":"020af691-e914-4355-8065-66ec1cc46c46","Type":"ContainerDied","Data":"a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0"}
Sep 16 13:51:52.510058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.509502    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5" event={"ID":"020af691-e914-4355-8065-66ec1cc46c46","Type":"ContainerDied","Data":"32fdfedcf3e9a3eacd6caaf2beb69be7d0831201ce6946bcf310ad05e48c9b39"}
Sep 16 13:51:52.510058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.509533    2556 scope.go:117] "RemoveContainer" containerID="a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0"
Sep 16 13:51:52.510058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.509741    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"
Sep 16 13:51:52.520827 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.520810    2556 scope.go:117] "RemoveContainer" containerID="a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0"
Sep 16 13:51:52.521161 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:51:52.521139    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0\": container with ID starting with a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0 not found: ID does not exist" containerID="a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0"
Sep 16 13:51:52.521223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.521171    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0"} err="failed to get container status \"a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0\": rpc error: code = NotFound desc = could not find container \"a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0\": container with ID starting with a88f0dfbdcdfba2c9c87cf6be5c8c492ffc99b02b146c11e70f7b1060de790d0 not found: ID does not exist"
Sep 16 13:51:52.527904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.527874    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"]
Sep 16 13:51:52.530732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.530709    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-6c7792/test-model-6c7792-kserve-747d895877-8p7b5"]
Sep 16 13:51:52.966072 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:52.966038    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:53.067265 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:53.067235    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:53.167263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:53.167232    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:53.233584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:53.233502    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:53.700894 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:53.700865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:54.059188 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:54.059074    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="020af691-e914-4355-8065-66ec1cc46c46" path="/var/lib/kubelet/pods/020af691-e914-4355-8065-66ec1cc46c46/volumes"
Sep 16 13:51:54.425645 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:54.425622    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:55.269911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:55.269881    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:56.012109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:56.012056    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:56.702507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:56.702473    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:57.373339 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:57.373308    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:58.047711 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.047672    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:58.174503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.174468    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:58.749939 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.749905    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:58.750171 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.749963    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:58.762385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.762362    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:51:58.794376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.794346    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7692-5dad7c"]
Sep 16 13:51:58.794803 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.794792    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="020af691-e914-4355-8065-66ec1cc46c46" containerName="main"
Sep 16 13:51:58.794849 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.794805    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="020af691-e914-4355-8065-66ec1cc46c46" containerName="main"
Sep 16 13:51:58.794899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.794890    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="020af691-e914-4355-8065-66ec1cc46c46" containerName="main"
Sep 16 13:51:58.798179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.798165    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:51:58.800035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.800015    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cmn8d\""
Sep 16 13:51:58.804070 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.804041    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-5dad7c"]
Sep 16 13:51:58.886406 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.886374    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:58.910712 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:58.910678    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctrs\" (UniqueName: \"kubernetes.io/projected/11f5c030-35c1-4ff9-87e4-710f3b6bfa23-kube-api-access-gctrs\") pod \"test-curl-7692-5dad7c\" (UID: \"11f5c030-35c1-4ff9-87e4-710f3b6bfa23\") " pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:51:59.011763 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.011665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctrs\" (UniqueName: \"kubernetes.io/projected/11f5c030-35c1-4ff9-87e4-710f3b6bfa23-kube-api-access-gctrs\") pod \"test-curl-7692-5dad7c\" (UID: \"11f5c030-35c1-4ff9-87e4-710f3b6bfa23\") " pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:51:59.019358 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.019320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctrs\" (UniqueName: \"kubernetes.io/projected/11f5c030-35c1-4ff9-87e4-710f3b6bfa23-kube-api-access-gctrs\") pod \"test-curl-7692-5dad7c\" (UID: \"11f5c030-35c1-4ff9-87e4-710f3b6bfa23\") " pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:51:59.108261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.108228    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:51:59.381591 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.381562    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7692-5dad7c"]
Sep 16 13:51:59.383264 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:51:59.383235    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11f5c030_35c1_4ff9_87e4_710f3b6bfa23.slice/crio-6dac09f43700f2cd25452a6f449ab13e69d3aaa2d631ab338d627698849a216b WatchSource:0}: Error finding container 6dac09f43700f2cd25452a6f449ab13e69d3aaa2d631ab338d627698849a216b: Status 404 returned error can't find the container with id 6dac09f43700f2cd25452a6f449ab13e69d3aaa2d631ab338d627698849a216b
Sep 16 13:51:59.385189 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.385168    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:51:59.555679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.555637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-5dad7c" event={"ID":"11f5c030-35c1-4ff9-87e4-710f3b6bfa23","Type":"ContainerStarted","Data":"6dac09f43700f2cd25452a6f449ab13e69d3aaa2d631ab338d627698849a216b"}
Sep 16 13:51:59.565485 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.565450    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:51:59.567817 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:51:59.567776    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:52:00.380556 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:00.380516    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:03.717022 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:03.716980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:04.369352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.369315    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"]
Sep 16 13:52:04.369725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.369694    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9" podUID="45273968-aaf2-4482-8f10-4c861316753b" containerName="maas-api" containerID="cri-o://2de7b0443772883bcbf4fe876159af19b60545e2afdf827425b99cac36efeac8" gracePeriod=30
Sep 16 13:52:04.415810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.415756    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:04.501106 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.500992    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:04.608883 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.608779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:04.617477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.614879    2556 generic.go:356] "Generic (PLEG): container finished" podID="45273968-aaf2-4482-8f10-4c861316753b" containerID="2de7b0443772883bcbf4fe876159af19b60545e2afdf827425b99cac36efeac8" exitCode=0
Sep 16 13:52:04.617477 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.615052    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9" event={"ID":"45273968-aaf2-4482-8f10-4c861316753b","Type":"ContainerDied","Data":"2de7b0443772883bcbf4fe876159af19b60545e2afdf827425b99cac36efeac8"}
Sep 16 13:52:04.664891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.664832    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:04.679313 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.679288    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:52:04.770770 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.770737    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-metrics-certs\") pod \"45273968-aaf2-4482-8f10-4c861316753b\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") "
Sep 16 13:52:04.771240 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.770849    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gb5j\" (UniqueName: \"kubernetes.io/projected/45273968-aaf2-4482-8f10-4c861316753b-kube-api-access-7gb5j\") pod \"45273968-aaf2-4482-8f10-4c861316753b\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") "
Sep 16 13:52:04.771240 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.770916    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-maas-api-tls\") pod \"45273968-aaf2-4482-8f10-4c861316753b\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") "
Sep 16 13:52:04.771240 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.770933    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45273968-aaf2-4482-8f10-4c861316753b-openshift-service-ca\") pod \"45273968-aaf2-4482-8f10-4c861316753b\" (UID: \"45273968-aaf2-4482-8f10-4c861316753b\") "
Sep 16 13:52:04.771434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.771397    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45273968-aaf2-4482-8f10-4c861316753b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45273968-aaf2-4482-8f10-4c861316753b" (UID: "45273968-aaf2-4482-8f10-4c861316753b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:52:04.773263 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.773239    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45273968-aaf2-4482-8f10-4c861316753b" (UID: "45273968-aaf2-4482-8f10-4c861316753b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:52:04.773372 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.773309    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45273968-aaf2-4482-8f10-4c861316753b" (UID: "45273968-aaf2-4482-8f10-4c861316753b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:52:04.773372 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.773335    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45273968-aaf2-4482-8f10-4c861316753b-kube-api-access-7gb5j" (OuterVolumeSpecName: "kube-api-access-7gb5j") pod "45273968-aaf2-4482-8f10-4c861316753b" (UID: "45273968-aaf2-4482-8f10-4c861316753b"). InnerVolumeSpecName "kube-api-access-7gb5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:52:04.872215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.872173    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gb5j\" (UniqueName: \"kubernetes.io/projected/45273968-aaf2-4482-8f10-4c861316753b-kube-api-access-7gb5j\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:52:04.872215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.872212    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:52:04.872215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.872222    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45273968-aaf2-4482-8f10-4c861316753b-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:52:04.872459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.872230    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45273968-aaf2-4482-8f10-4c861316753b-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:52:04.947614 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:04.947576    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:05.062307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.062263    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:05.622844 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.622817    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"
Sep 16 13:52:05.623052 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.622817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9" event={"ID":"45273968-aaf2-4482-8f10-4c861316753b","Type":"ContainerDied","Data":"71703a1934a763dd1de49d39ca086d619cc8ce979abc3f6f04e4c666106528af"}
Sep 16 13:52:05.623052 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.622959    2556 scope.go:117] "RemoveContainer" containerID="2de7b0443772883bcbf4fe876159af19b60545e2afdf827425b99cac36efeac8"
Sep 16 13:52:05.646973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.646942    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"]
Sep 16 13:52:05.652031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.651999    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-6c7792-7d486864b7-4n5g9"]
Sep 16 13:52:05.828044 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:05.828001    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:06.059433 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:06.059399    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45273968-aaf2-4482-8f10-4c861316753b" path="/var/lib/kubelet/pods/45273968-aaf2-4482-8f10-4c861316753b/volumes"
Sep 16 13:52:06.649510 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:06.649472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:07.369009 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:07.368973    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:07.470646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:07.470610    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:07.571252 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:07.571209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:08.050815 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:08.050766    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:08.769361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:08.769324    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:10.657371 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:10.657338    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:10.729700 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:10.729651    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:10.931836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:10.931799    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:11.487886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:11.487800    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:11.666215 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:11.666172    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7692-5dad7c" event={"ID":"11f5c030-35c1-4ff9-87e4-710f3b6bfa23","Type":"ContainerStarted","Data":"57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3"}
Sep 16 13:52:11.683978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:11.683926    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7692-5dad7c" podStartSLOduration=1.9458249890000001 podStartE2EDuration="13.683911474s" totalImagesPullingTime="11.738086485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:11.683323225 +0000 UTC m=+1142.149260138" watchObservedRunningTime="2026-09-16 13:52:11.683911474 +0000 UTC m=+1142.149848382"
Sep 16 13:52:11.807552 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:11.807460    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:11.907702 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:11.907662    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:12.343938 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.343901    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7692-5dad7c"]
Sep 16 13:52:12.347887 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.347863    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7692-5dad7c"]
Sep 16 13:52:12.673929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.673898    2556 generic.go:356] "Generic (PLEG): container finished" podID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerID="164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c" exitCode=0
Sep 16 13:52:12.674454 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.673977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" event={"ID":"052a453e-4175-4b96-8d9e-9ef0da349fe9","Type":"ContainerDied","Data":"164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"}
Sep 16 13:52:12.674454 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:52:12.674240    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7692-5dad7c" container="test-curl-7692-5dad7c"
Sep 16 13:52:12.674574 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.674525    2556 scope.go:117] "RemoveContainer" containerID="164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"
Sep 16 13:52:12.675651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.675621    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:12.675715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.675698    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:12.675749 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:12.675734    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:13.437886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:13.437851    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:52:13.682012 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:13.681978    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" event={"ID":"052a453e-4175-4b96-8d9e-9ef0da349fe9","Type":"ContainerStarted","Data":"c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b"}
Sep 16 13:52:13.682503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:13.682193    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7692-5dad7c" podUID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" containerName="test-curl-7692-5dad7c" containerID="cri-o://57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3" gracePeriod=2
Sep 16 13:52:13.706188 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:52:13.706064    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7692-5dad7c\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:52:13.856495 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:13.856458    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:52:13.857789 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:13.857762    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 13:52:13.857941 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:13.857816    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 13:52:14.801272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:14.801237    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:14.856233 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:14.856201    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 13:52:14.856410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:14.856288    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 13:52:15.400452 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:15.400410    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:15.497161 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:15.497128    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:15.681184 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:15.681147    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:15.855979 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:15.855941    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 13:52:15.856447 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:15.856025    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 13:52:15.920279 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:15.920254    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:52:16.001914 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.000927    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gctrs\" (UniqueName: \"kubernetes.io/projected/11f5c030-35c1-4ff9-87e4-710f3b6bfa23-kube-api-access-gctrs\") pod \"11f5c030-35c1-4ff9-87e4-710f3b6bfa23\" (UID: \"11f5c030-35c1-4ff9-87e4-710f3b6bfa23\") "
Sep 16 13:52:16.012640 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.012601    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f5c030-35c1-4ff9-87e4-710f3b6bfa23-kube-api-access-gctrs" (OuterVolumeSpecName: "kube-api-access-gctrs") pod "11f5c030-35c1-4ff9-87e4-710f3b6bfa23" (UID: "11f5c030-35c1-4ff9-87e4-710f3b6bfa23"). InnerVolumeSpecName "kube-api-access-gctrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:52:16.059810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.059777    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" path="/var/lib/kubelet/pods/11f5c030-35c1-4ff9-87e4-710f3b6bfa23/volumes"
Sep 16 13:52:16.103073 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.103032    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctrs\" (UniqueName: \"kubernetes.io/projected/11f5c030-35c1-4ff9-87e4-710f3b6bfa23-kube-api-access-gctrs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:52:16.492663 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.492622    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:16.714079 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.714046    2556 generic.go:356] "Generic (PLEG): container finished" podID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" containerID="57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3" exitCode=137
Sep 16 13:52:16.714686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.714112    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7692-5dad7c"
Sep 16 13:52:16.714686 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.714134    2556 scope.go:117] "RemoveContainer" containerID="57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3"
Sep 16 13:52:16.724430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.724407    2556 scope.go:117] "RemoveContainer" containerID="57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3"
Sep 16 13:52:16.724707 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:52:16.724688    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3\": container with ID starting with 57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3 not found: ID does not exist" containerID="57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3"
Sep 16 13:52:16.724773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.724716    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3"} err="failed to get container status \"57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3\": rpc error: code = NotFound desc = could not find container \"57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3\": container with ID starting with 57986f3ccb7407578177b0034c26ba7486df5ff86e9aed469357b019e7796bf3 not found: ID does not exist"
Sep 16 13:52:16.856221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.856131    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 13:52:16.856221 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:16.856183    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 13:52:17.182011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:17.181980    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:17.856590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:17.856555    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 16 13:52:17.856973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:17.856620    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 16 13:52:18.101626 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.101588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:18.795616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.795548    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:18.854179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.854108    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:18.877573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.877528    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:52:18.878083 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.877952    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:52:18.879354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.879332    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:52:18.984725 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:18.984696    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:19.046621 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.046542    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:52:19.085738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.085710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:19.400875 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.400786    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"]
Sep 16 13:52:19.401717 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.401690    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45273968-aaf2-4482-8f10-4c861316753b" containerName="maas-api"
Sep 16 13:52:19.401944 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.401879    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="45273968-aaf2-4482-8f10-4c861316753b" containerName="maas-api"
Sep 16 13:52:19.402371 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.402354    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" containerName="test-curl-7692-5dad7c"
Sep 16 13:52:19.402487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.402477    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" containerName="test-curl-7692-5dad7c"
Sep 16 13:52:19.402768 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.402754    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11f5c030-35c1-4ff9-87e4-710f3b6bfa23" containerName="test-curl-7692-5dad7c"
Sep 16 13:52:19.402862 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.402852    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="45273968-aaf2-4482-8f10-4c861316753b" containerName="maas-api"
Sep 16 13:52:19.411929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.410942    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.416373 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.416345    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3be042d3-openshift-default-dockercfg-cpn6v\""
Sep 16 13:52:19.417320 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.417298    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"]
Sep 16 13:52:19.434923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.434895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:19.534822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.534783    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:19.543960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.543929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-podinfo\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.543989    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-envoy\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544052    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istiod-ca-cert\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544154    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-kube-api-access-pstlf\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544205    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-data\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544377 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544307    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-certs\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544377 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544329    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-token\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544475 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-credential-socket\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.544475 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.544459    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-socket\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.619009 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.618970    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:19.645675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-envoy\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.645851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istiod-ca-cert\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.645851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645729    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-kube-api-access-pstlf\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.645851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645757    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-data\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.645851 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645839    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-certs\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645864    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-token\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645920    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-credential-socket\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.645968    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-socket\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646081 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.646001    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-podinfo\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.646193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-data\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.646264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-certs\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.646534    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istiod-ca-cert\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.646599    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-credential-socket\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.646742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.646686    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-socket\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.648170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.648151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-envoy\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.648560 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.648537    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-podinfo\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.653900 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.653852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-token\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.654101 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.654064    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-kube-api-access-pstlf\") pod \"e2e-mt-3be042d3-openshift-default-6495657988-9cj8n\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") " pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:19.681634 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.681608    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:19.731176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:19.731144    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:20.016138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.016076    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"]
Sep 16 13:52:20.016852 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:52:20.016827    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3762dc_7d18_4eba_ad8d_3f06dc1dba62.slice/crio-7c435d12b5ee6d046a51e7f3b73c046bc55251a4b0ce161f143151b51bc63879 WatchSource:0}: Error finding container 7c435d12b5ee6d046a51e7f3b73c046bc55251a4b0ce161f143151b51bc63879: Status 404 returned error can't find the container with id 7c435d12b5ee6d046a51e7f3b73c046bc55251a4b0ce161f143151b51bc63879
Sep 16 13:52:20.019738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.019703    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:20.019820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.019779    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:20.019820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.019810    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:20.750232 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.750194    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" event={"ID":"af3762dc-7d18-4eba-ad8d-3f06dc1dba62","Type":"ContainerStarted","Data":"81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"}
Sep 16 13:52:20.750397 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.750259    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" event={"ID":"af3762dc-7d18-4eba-ad8d-3f06dc1dba62","Type":"ContainerStarted","Data":"7c435d12b5ee6d046a51e7f3b73c046bc55251a4b0ce161f143151b51bc63879"}
Sep 16 13:52:20.767017 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:20.766962    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podStartSLOduration=1.766946176 podStartE2EDuration="1.766946176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:20.765723179 +0000 UTC m=+1151.231660087" watchObservedRunningTime="2026-09-16 13:52:20.766946176 +0000 UTC m=+1151.232883084"
Sep 16 13:52:21.732311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:21.732269    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:21.737962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:21.737935    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:21.756663 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:21.756627    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:21.758171 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:21.758147    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:22.517432 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:22.517383    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:22.706541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:22.706508    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:22.801546 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:22.801468    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:23.340353 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:23.340314    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:24.180469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:24.180433    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:25.060028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:25.059993    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:25.714723 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:25.714690    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:26.476859 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:26.476823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:27.189046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:27.189012    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:27.287481 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:27.287447    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:27.387751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:27.387720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:27.643301 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:27.643225    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:27.756332 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:27.756296    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:28.043574 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:28.043537    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:28.760502 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:28.760470    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:29.580389 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:29.580359    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:30.129611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.129571    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:30.257804 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.257767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:30.468953 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.468918    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:30.744678 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.744586    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"]
Sep 16 13:52:30.749884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.749857    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.752233 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.751992    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ac8045-openshift-default-dockercfg-tnphd\""
Sep 16 13:52:30.754194 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.754160    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"]
Sep 16 13:52:30.761746 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.761707    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:30.857659 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.857618    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:30.863971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.863936    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2b0e93-9314-404f-b55e-2118d4b028eb-istiod-ca-cert\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.863995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-certs\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-podinfo\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-envoy\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m526d\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-kube-api-access-m526d\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-token\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864298    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-data\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864375    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-socket\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.864448 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.864414    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-credential-socket\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.965589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965548    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-certs\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.965800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-podinfo\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.965800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-envoy\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.965800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965694    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m526d\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-kube-api-access-m526d\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.965800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965729    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-token\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.965800 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-data\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-socket\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965887    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-credential-socket\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.965966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2b0e93-9314-404f-b55e-2118d4b028eb-istiod-ca-cert\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.966036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-certs\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.966289    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-data\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.966312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-socket\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966437 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.966395    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-credential-socket\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.966602 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.966582    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2b0e93-9314-404f-b55e-2118d4b028eb-istiod-ca-cert\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.968710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.968683    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-envoy\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.968845 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.968828    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-podinfo\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.973538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.973512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-token\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:30.973639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:30.973613    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m526d\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-kube-api-access-m526d\") pod \"e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") " pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:31.063344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.063255    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:31.084642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.084590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:31.345174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.345144    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"]
Sep 16 13:52:31.345523 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:52:31.345400    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2b0e93_9314_404f_b55e_2118d4b028eb.slice/crio-5e690e7a9472fb44af4e77a558d1013b4972ce04f367732321ae634083b5c8f4 WatchSource:0}: Error finding container 5e690e7a9472fb44af4e77a558d1013b4972ce04f367732321ae634083b5c8f4: Status 404 returned error can't find the container with id 5e690e7a9472fb44af4e77a558d1013b4972ce04f367732321ae634083b5c8f4
Sep 16 13:52:31.347770 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.347737    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:31.347859 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.347801    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:31.347859 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.347828    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:31.772248 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.772213    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:31.822354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.822321    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" event={"ID":"1e2b0e93-9314-404f-b55e-2118d4b028eb","Type":"ContainerStarted","Data":"35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"}
Sep 16 13:52:31.822557 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.822374    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" event={"ID":"1e2b0e93-9314-404f-b55e-2118d4b028eb","Type":"ContainerStarted","Data":"5e690e7a9472fb44af4e77a558d1013b4972ce04f367732321ae634083b5c8f4"}
Sep 16 13:52:31.846011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:31.845966    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podStartSLOduration=1.845950918 podStartE2EDuration="1.845950918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:31.84255937 +0000 UTC m=+1162.308496277" watchObservedRunningTime="2026-09-16 13:52:31.845950918 +0000 UTC m=+1162.311888050"
Sep 16 13:52:32.063435 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:32.063361    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:32.068757 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:32.068731    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:32.572867 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:32.572829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:32.829186 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:32.829070    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:32.830212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:32.830195    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:52:33.274643 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:33.274602    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:33.761251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:33.761218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:33.854818 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:33.854786    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:34.089015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:34.088936    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:34.800692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:34.800657    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:35.635646 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.635612    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"]
Sep 16 13:52:35.639517 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.639498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.641493 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.641475    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d96c7\"/\"distinct-2-w0-7d96c7-kserve-self-signed-certs\""
Sep 16 13:52:35.646024 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.645997    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"]
Sep 16 13:52:35.665861 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.665837    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:35.713996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.713962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-home\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.713996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.713999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae901fb1-999e-4291-999d-bdfa6d479ded-tls-certs\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.714242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.714141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-model-cache\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.714242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.714186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mntz\" (UniqueName: \"kubernetes.io/projected/ae901fb1-999e-4291-999d-bdfa6d479ded-kube-api-access-7mntz\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.714242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.714209    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-tmp-dir\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.714242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.714226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-dshm\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.815497 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815463    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-home\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.815497 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae901fb1-999e-4291-999d-bdfa6d479ded-tls-certs\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-model-cache\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mntz\" (UniqueName: \"kubernetes.io/projected/ae901fb1-999e-4291-999d-bdfa6d479ded-kube-api-access-7mntz\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-tmp-dir\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-dshm\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.815899    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-home\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.816196    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-model-cache\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.816336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.816285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-tmp-dir\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.818038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.818018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-dshm\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.818814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.818796    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae901fb1-999e-4291-999d-bdfa6d479ded-tls-certs\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.825039 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.825012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mntz\" (UniqueName: \"kubernetes.io/projected/ae901fb1-999e-4291-999d-bdfa6d479ded-kube-api-access-7mntz\") pod \"distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") " pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:35.844773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.844742    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:35.950559 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:35.950516    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:36.230409 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:36.229176    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"]
Sep 16 13:52:36.234544 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:52:36.234503    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae901fb1_999e_4291_999d_bdfa6d479ded.slice/crio-d964a9f7b28c00d144cb578d880717033ccad91307439a6c44ef338097968739 WatchSource:0}: Error finding container d964a9f7b28c00d144cb578d880717033ccad91307439a6c44ef338097968739: Status 404 returned error can't find the container with id d964a9f7b28c00d144cb578d880717033ccad91307439a6c44ef338097968739
Sep 16 13:52:36.378831 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:36.378790    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:36.857672 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:36.857637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2" event={"ID":"ae901fb1-999e-4291-999d-bdfa6d479ded","Type":"ContainerStarted","Data":"98441fb39a0cb6f6ec85163c8e42696b60835a6170f1f7f85a84f4c6b3e26d19"}
Sep 16 13:52:36.858180 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:36.857723    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2" event={"ID":"ae901fb1-999e-4291-999d-bdfa6d479ded","Type":"ContainerStarted","Data":"d964a9f7b28c00d144cb578d880717033ccad91307439a6c44ef338097968739"}
Sep 16 13:52:36.874642 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:36.874589    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2" podStartSLOduration=1.8745722 podStartE2EDuration="1.8745722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:52:36.872830858 +0000 UTC m=+1167.338767777" watchObservedRunningTime="2026-09-16 13:52:36.8745722 +0000 UTC m=+1167.340509108"
Sep 16 13:52:37.117958 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:37.117869    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:37.787358 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:37.787320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:38.723558 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:38.723517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:38.814451 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:38.814398    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:38.914607 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:38.914570    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:39.240494 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:39.240461    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:39.340830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:39.340791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:39.877160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:39.877127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:40.673798 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:40.673763    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:41.373890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:41.373857    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:42.191584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:42.191555    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:42.588227 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:42.588195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:42.886449 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:42.886357    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:43.545143 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:43.545076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:44.392853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:44.392819    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:45.039942 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.039902    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:45.924345 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.924311    2556 generic.go:356] "Generic (PLEG): container finished" podID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerID="81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec" exitCode=0
Sep 16 13:52:45.924809 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.924396    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" event={"ID":"af3762dc-7d18-4eba-ad8d-3f06dc1dba62","Type":"ContainerDied","Data":"81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"}
Sep 16 13:52:45.924937 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.924921    2556 scope.go:117] "RemoveContainer" containerID="81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"
Sep 16 13:52:45.926151 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.926124    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:45.926217 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.926189    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:45.926254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.926222    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:52:45.951442 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.951410    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:45.951723 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.951706    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:45.968660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:45.967976    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:46.933491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:46.933458    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" event={"ID":"af3762dc-7d18-4eba-ad8d-3f06dc1dba62","Type":"ContainerStarted","Data":"28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf"}
Sep 16 13:52:46.945130 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:46.945103    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:52:47.731524 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:47.731483    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:47.732835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:47.732808    2556 patch_prober.go:28] interesting pod/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 13:52:47.732986 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:47.732860    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 13:52:48.180307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:48.180274    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:48.732008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:48.731967    2556 patch_prober.go:28] interesting pod/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 13:52:48.732229 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:48.732025    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 13:52:49.732192 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:49.732156    2556 patch_prober.go:28] interesting pod/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 13:52:49.732588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:49.732226    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 13:52:50.732228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:50.732198    2556 patch_prober.go:28] interesting pod/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 13:52:50.732599 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:50.732248    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 13:52:51.732410 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:51.732369    2556 patch_prober.go:28] interesting pod/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 16 13:52:51.732822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:51.732440    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 16 13:52:51.756584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:51.756541    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:52.736275 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:52.736246    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:52.737668 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:52.737645    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:52:55.272208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:55.272155    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:52:56.046625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:56.046594    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:52:59.623911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:52:59.623865    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:01.046036 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:01.045997    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:01.702917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:01.702878    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:02.371352 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:02.371319    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:03.091883 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:03.091842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:03.748667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:03.748634    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:03.859766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:03.859728    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:04.638766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:04.638729    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:05.367734 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:05.367700    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:06.015768 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:06.015725    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:10.107005 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:10.106972    2556 generic.go:356] "Generic (PLEG): container finished" podID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerID="35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac" exitCode=0
Sep 16 13:53:10.107549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:10.107031    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" event={"ID":"1e2b0e93-9314-404f-b55e-2118d4b028eb","Type":"ContainerDied","Data":"35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"}
Sep 16 13:53:10.107615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:10.107590    2556 scope.go:117] "RemoveContainer" containerID="35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"
Sep 16 13:53:10.109040 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:10.109007    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:53:10.109200 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:10.109107    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:53:10.109200 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:10.109159    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:53:11.118722 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:11.118690    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" event={"ID":"1e2b0e93-9314-404f-b55e-2118d4b028eb","Type":"ContainerStarted","Data":"461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5"}
Sep 16 13:53:12.063917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:12.063882    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:53:12.064235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:12.064108    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 13:53:12.064235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:12.064151    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 13:53:13.063928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:13.063894    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 13:53:13.064337 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:13.063947    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 13:53:14.063807 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:14.063768    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 13:53:14.063996 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:14.063836    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 13:53:15.063821 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:15.063780    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 13:53:15.064010 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:15.063838    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 13:53:16.064576 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:16.064543    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 16 13:53:16.064960 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:16.064602    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 16 13:53:17.067475 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:17.067441    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:53:17.067911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:17.067887    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:53:17.068632 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:17.068611    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 13:53:25.239242 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.239200    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"]
Sep 16 13:53:25.243340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.243307    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.245956 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.245930    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-dffccf\"/\"openshift-service-ca.crt\""
Sep 16 13:53:25.245956 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.245961    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-dffccf\"/\"noten-model-dffccf-kserve-self-signed-certs\""
Sep 16 13:53:25.246210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.245991    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-dffccf\"/\"kube-root-ca.crt\""
Sep 16 13:53:25.246210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.245929    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-dffccf\"/\"default-dockercfg-cwv5m\""
Sep 16 13:53:25.255487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.255464    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"]
Sep 16 13:53:25.274193 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.274159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:25.371719 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.371684    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f96m\" (UniqueName: \"kubernetes.io/projected/081686fa-4b02-41cc-af6b-4416ba0713bf-kube-api-access-5f96m\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.371855 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.371729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-tmp-dir\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.371855 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.371779    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-model-cache\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.371855 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.371817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-home\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.371855 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.371851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081686fa-4b02-41cc-af6b-4416ba0713bf-tls-certs\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.372029 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.371881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-dshm\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.472853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.472804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f96m\" (UniqueName: \"kubernetes.io/projected/081686fa-4b02-41cc-af6b-4416ba0713bf-kube-api-access-5f96m\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.472853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.472860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-tmp-dir\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473148 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.472899    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-model-cache\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473148 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.472927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-home\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473148 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.472965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081686fa-4b02-41cc-af6b-4416ba0713bf-tls-certs\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473148 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.473011    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-dshm\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473429 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.473378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-tmp-dir\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473544 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.473392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-home\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.473544 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.473475    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-model-cache\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.475430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.475408    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-dshm\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.475744 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.475728    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081686fa-4b02-41cc-af6b-4416ba0713bf-tls-certs\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.480381 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.480364    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f96m\" (UniqueName: \"kubernetes.io/projected/081686fa-4b02-41cc-af6b-4416ba0713bf-kube-api-access-5f96m\") pod \"noten-model-dffccf-kserve-7d959fcb75-xg8dw\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") " pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.554040 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.553964    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:25.831564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:25.831539    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"]
Sep 16 13:53:25.832421 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:53:25.832396    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod081686fa_4b02_41cc_af6b_4416ba0713bf.slice/crio-38104748901ce9c670eb3ed6291e4093a31fb6711ce880708eca9428f5f83ed2 WatchSource:0}: Error finding container 38104748901ce9c670eb3ed6291e4093a31fb6711ce880708eca9428f5f83ed2: Status 404 returned error can't find the container with id 38104748901ce9c670eb3ed6291e4093a31fb6711ce880708eca9428f5f83ed2
Sep 16 13:53:26.227928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:26.227894    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw" event={"ID":"081686fa-4b02-41cc-af6b-4416ba0713bf","Type":"ContainerStarted","Data":"62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634"}
Sep 16 13:53:26.227928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:26.227935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw" event={"ID":"081686fa-4b02-41cc-af6b-4416ba0713bf","Type":"ContainerStarted","Data":"38104748901ce9c670eb3ed6291e4093a31fb6711ce880708eca9428f5f83ed2"}
Sep 16 13:53:26.244224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:26.244173    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw" podStartSLOduration=1.244158042 podStartE2EDuration="1.244158042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:26.242476641 +0000 UTC m=+1216.708413566" watchObservedRunningTime="2026-09-16 13:53:26.244158042 +0000 UTC m=+1216.710094949"
Sep 16 13:53:31.363708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:31.363675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:32.496238 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:32.496204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:35.554357 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:35.554324    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:35.554746 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:35.554400    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:35.567178 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:35.567152    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:36.303431 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:36.303401    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:53:52.049990 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.049953    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"]
Sep 16 13:53:52.054578 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.054551    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.058206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.057394    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3be042d3\"/\"e2e-lifecycle-model-3be042d3-kserve-self-signed-certs\""
Sep 16 13:53:52.058206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.057645    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3be042d3\"/\"kube-root-ca.crt\""
Sep 16 13:53:52.058206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.057852    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3be042d3\"/\"default-dockercfg-8mlwc\""
Sep 16 13:53:52.058206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.058019    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3be042d3\"/\"openshift-service-ca.crt\""
Sep 16 13:53:52.065660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.065639    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"]
Sep 16 13:53:52.078756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.078720    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:53:52.165379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.165338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-home\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.165379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.165377    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tmp-dir\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.165648 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.165542    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tls-certs\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.165863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.165829    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-model-cache\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.166013 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.165958    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76282\" (UniqueName: \"kubernetes.io/projected/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-kube-api-access-76282\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.166132 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.166020    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-dshm\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.267814 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.267769    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tls-certs\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.267846    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-model-cache\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.267880    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76282\" (UniqueName: \"kubernetes.io/projected/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-kube-api-access-76282\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.267910    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-dshm\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.267986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-home\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.268000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tmp-dir\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.268338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-model-cache\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.268370    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tmp-dir\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.268440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.268411    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-home\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.270635 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.270610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-dshm\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.270790 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.270772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tls-certs\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.276673 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.276643    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76282\" (UniqueName: \"kubernetes.io/projected/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-kube-api-access-76282\") pod \"e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") " pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.369299 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.369210    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:53:52.647958 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:52.647908    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"]
Sep 16 13:53:52.650740 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:53:52.650705    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcde9bb33_7d27_4f67_9d3b_3643e2a56ffd.slice/crio-d29f964709f6fa27c73dee1c4768222f8108105540e88d0044fd04f147bed7fb WatchSource:0}: Error finding container d29f964709f6fa27c73dee1c4768222f8108105540e88d0044fd04f147bed7fb: Status 404 returned error can't find the container with id d29f964709f6fa27c73dee1c4768222f8108105540e88d0044fd04f147bed7fb
Sep 16 13:53:53.403868 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:53.403834    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s" event={"ID":"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd","Type":"ContainerStarted","Data":"6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6"}
Sep 16 13:53:53.404254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:53.403875    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s" event={"ID":"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd","Type":"ContainerStarted","Data":"d29f964709f6fa27c73dee1c4768222f8108105540e88d0044fd04f147bed7fb"}
Sep 16 13:53:53.440656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:53:53.440609    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s" podStartSLOduration=1.440592796 podStartE2EDuration="1.440592796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:53:53.437758024 +0000 UTC m=+1243.903694952" watchObservedRunningTime="2026-09-16 13:53:53.440592796 +0000 UTC m=+1243.906529723"
Sep 16 13:54:00.600706 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:00.600670    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:00.814696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:00.814658    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:00.925307 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:00.925271    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:02.369789 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:02.369750    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:54:02.370233 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:02.369832    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:54:02.382644 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:02.382611    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:54:02.473260 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:02.473226    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:54:04.620210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:04.620176    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:04.669597 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:04.669565    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:05.909311 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:05.909271    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"]
Sep 16 13:54:05.947863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:05.947827    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:06.081309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:06.081282    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:06.512476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:06.512446    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:06.614376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:06.614344    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:08.725738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:08.725704    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"]
Sep 16 13:54:08.742564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:08.742540    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:10.638911 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:10.638876    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:12.262412 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:12.262378    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"]
Sep 16 13:54:12.321957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:12.321919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:19.151803 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:19.151766    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:54:20.930865 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:20.930815    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw" podUID="081686fa-4b02-41cc-af6b-4416ba0713bf" containerName="main" containerID="cri-o://62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634" gracePeriod=45
Sep 16 13:54:21.196359 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.196296    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:54:21.278159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278120    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-home\") pod \"081686fa-4b02-41cc-af6b-4416ba0713bf\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") "
Sep 16 13:54:21.278159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278166    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081686fa-4b02-41cc-af6b-4416ba0713bf-tls-certs\") pod \"081686fa-4b02-41cc-af6b-4416ba0713bf\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") "
Sep 16 13:54:21.278422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278235    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f96m\" (UniqueName: \"kubernetes.io/projected/081686fa-4b02-41cc-af6b-4416ba0713bf-kube-api-access-5f96m\") pod \"081686fa-4b02-41cc-af6b-4416ba0713bf\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") "
Sep 16 13:54:21.278422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278316    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-tmp-dir\") pod \"081686fa-4b02-41cc-af6b-4416ba0713bf\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") "
Sep 16 13:54:21.278422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278370    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-model-cache\") pod \"081686fa-4b02-41cc-af6b-4416ba0713bf\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") "
Sep 16 13:54:21.278422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278379    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-home" (OuterVolumeSpecName: "home") pod "081686fa-4b02-41cc-af6b-4416ba0713bf" (UID: "081686fa-4b02-41cc-af6b-4416ba0713bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:21.278422 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278393    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-dshm\") pod \"081686fa-4b02-41cc-af6b-4416ba0713bf\" (UID: \"081686fa-4b02-41cc-af6b-4416ba0713bf\") "
Sep 16 13:54:21.278692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278616    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "081686fa-4b02-41cc-af6b-4416ba0713bf" (UID: "081686fa-4b02-41cc-af6b-4416ba0713bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:21.278692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278679    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "081686fa-4b02-41cc-af6b-4416ba0713bf" (UID: "081686fa-4b02-41cc-af6b-4416ba0713bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:21.279059 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.278997    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:21.279059 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.279022    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:21.279059 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.279035    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:21.280782 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.280750    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-dshm" (OuterVolumeSpecName: "dshm") pod "081686fa-4b02-41cc-af6b-4416ba0713bf" (UID: "081686fa-4b02-41cc-af6b-4416ba0713bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:21.280782 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.280761    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081686fa-4b02-41cc-af6b-4416ba0713bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "081686fa-4b02-41cc-af6b-4416ba0713bf" (UID: "081686fa-4b02-41cc-af6b-4416ba0713bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:54:21.280959 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.280817    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081686fa-4b02-41cc-af6b-4416ba0713bf-kube-api-access-5f96m" (OuterVolumeSpecName: "kube-api-access-5f96m") pod "081686fa-4b02-41cc-af6b-4416ba0713bf" (UID: "081686fa-4b02-41cc-af6b-4416ba0713bf"). InnerVolumeSpecName "kube-api-access-5f96m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:21.380272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.380232    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/081686fa-4b02-41cc-af6b-4416ba0713bf-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:21.380272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.380273    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/081686fa-4b02-41cc-af6b-4416ba0713bf-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:21.380480 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.380291    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f96m\" (UniqueName: \"kubernetes.io/projected/081686fa-4b02-41cc-af6b-4416ba0713bf-kube-api-access-5f96m\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:21.585587 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.585490    2556 generic.go:356] "Generic (PLEG): container finished" podID="081686fa-4b02-41cc-af6b-4416ba0713bf" containerID="62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634" exitCode=0
Sep 16 13:54:21.585587 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.585528    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"
Sep 16 13:54:21.585587 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.585571    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw" event={"ID":"081686fa-4b02-41cc-af6b-4416ba0713bf","Type":"ContainerDied","Data":"62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634"}
Sep 16 13:54:21.585870 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.585616    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw" event={"ID":"081686fa-4b02-41cc-af6b-4416ba0713bf","Type":"ContainerDied","Data":"38104748901ce9c670eb3ed6291e4093a31fb6711ce880708eca9428f5f83ed2"}
Sep 16 13:54:21.585870 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.585638    2556 scope.go:117] "RemoveContainer" containerID="62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634"
Sep 16 13:54:21.600076 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.600049    2556 scope.go:117] "RemoveContainer" containerID="62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634"
Sep 16 13:54:21.600420 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:54:21.600398    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634\": container with ID starting with 62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634 not found: ID does not exist" containerID="62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634"
Sep 16 13:54:21.600487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.600449    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634"} err="failed to get container status \"62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634\": rpc error: code = NotFound desc = could not find container \"62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634\": container with ID starting with 62922869fa52c96dd46733fa9a8e447cffdfb2272203ae2c9bc4f88ee048d634 not found: ID does not exist"
Sep 16 13:54:21.611296 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.611274    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"]
Sep 16 13:54:21.614008 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:21.613988    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-dffccf/noten-model-dffccf-kserve-7d959fcb75-xg8dw"]
Sep 16 13:54:22.060414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:22.060380    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="081686fa-4b02-41cc-af6b-4416ba0713bf" path="/var/lib/kubelet/pods/081686fa-4b02-41cc-af6b-4416ba0713bf/volumes"
Sep 16 13:54:23.748246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:23.748182    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv" podUID="cf5c8c5a-8da8-44de-887c-5dd8af358572" containerName="main" containerID="cri-o://2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4" gracePeriod=45
Sep 16 13:54:24.005685 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.005613    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:54:24.106953 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.106912    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5c8c5a-8da8-44de-887c-5dd8af358572-tls-certs\") pod \"cf5c8c5a-8da8-44de-887c-5dd8af358572\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") "
Sep 16 13:54:24.107181 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107054    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-tmp-dir\") pod \"cf5c8c5a-8da8-44de-887c-5dd8af358572\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") "
Sep 16 13:54:24.107181 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107119    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-dshm\") pod \"cf5c8c5a-8da8-44de-887c-5dd8af358572\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") "
Sep 16 13:54:24.107181 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107176    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-model-cache\") pod \"cf5c8c5a-8da8-44de-887c-5dd8af358572\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") "
Sep 16 13:54:24.107361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107237    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqz8k\" (UniqueName: \"kubernetes.io/projected/cf5c8c5a-8da8-44de-887c-5dd8af358572-kube-api-access-kqz8k\") pod \"cf5c8c5a-8da8-44de-887c-5dd8af358572\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") "
Sep 16 13:54:24.107361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107278    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-home\") pod \"cf5c8c5a-8da8-44de-887c-5dd8af358572\" (UID: \"cf5c8c5a-8da8-44de-887c-5dd8af358572\") "
Sep 16 13:54:24.107361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107329    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cf5c8c5a-8da8-44de-887c-5dd8af358572" (UID: "cf5c8c5a-8da8-44de-887c-5dd8af358572"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:24.107523 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107428    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-model-cache" (OuterVolumeSpecName: "model-cache") pod "cf5c8c5a-8da8-44de-887c-5dd8af358572" (UID: "cf5c8c5a-8da8-44de-887c-5dd8af358572"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:24.107636 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107579    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-home" (OuterVolumeSpecName: "home") pod "cf5c8c5a-8da8-44de-887c-5dd8af358572" (UID: "cf5c8c5a-8da8-44de-887c-5dd8af358572"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:24.107914 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107891    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:24.107970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107918    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:24.107970 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.107933    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:24.109382 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.109361    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-dshm" (OuterVolumeSpecName: "dshm") pod "cf5c8c5a-8da8-44de-887c-5dd8af358572" (UID: "cf5c8c5a-8da8-44de-887c-5dd8af358572"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:24.109472 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.109406    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5c8c5a-8da8-44de-887c-5dd8af358572-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf5c8c5a-8da8-44de-887c-5dd8af358572" (UID: "cf5c8c5a-8da8-44de-887c-5dd8af358572"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:54:24.109522 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.109505    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5c8c5a-8da8-44de-887c-5dd8af358572-kube-api-access-kqz8k" (OuterVolumeSpecName: "kube-api-access-kqz8k") pod "cf5c8c5a-8da8-44de-887c-5dd8af358572" (UID: "cf5c8c5a-8da8-44de-887c-5dd8af358572"). InnerVolumeSpecName "kube-api-access-kqz8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:24.209021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.208979    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf5c8c5a-8da8-44de-887c-5dd8af358572-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:24.209021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.209018    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqz8k\" (UniqueName: \"kubernetes.io/projected/cf5c8c5a-8da8-44de-887c-5dd8af358572-kube-api-access-kqz8k\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:24.209265 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.209033    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf5c8c5a-8da8-44de-887c-5dd8af358572-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:24.608808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.608774    2556 generic.go:356] "Generic (PLEG): container finished" podID="cf5c8c5a-8da8-44de-887c-5dd8af358572" containerID="2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4" exitCode=0
Sep 16 13:54:24.608976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.608827    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"
Sep 16 13:54:24.608976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.608865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv" event={"ID":"cf5c8c5a-8da8-44de-887c-5dd8af358572","Type":"ContainerDied","Data":"2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4"}
Sep 16 13:54:24.608976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.608912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv" event={"ID":"cf5c8c5a-8da8-44de-887c-5dd8af358572","Type":"ContainerDied","Data":"3a9966d8b5b8346c0dd571c9c47c0065ccddcc86dcf28ac4e253d25c31cdf0e0"}
Sep 16 13:54:24.608976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.608932    2556 scope.go:117] "RemoveContainer" containerID="2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4"
Sep 16 13:54:24.619570 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.619552    2556 scope.go:117] "RemoveContainer" containerID="2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4"
Sep 16 13:54:24.619823 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:54:24.619804    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4\": container with ID starting with 2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4 not found: ID does not exist" containerID="2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4"
Sep 16 13:54:24.619869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.619831    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4"} err="failed to get container status \"2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4\": rpc error: code = NotFound desc = could not find container \"2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4\": container with ID starting with 2e066f4bd3e91efa354cc6af26c77b1d7042a1be71381f73542d7ad24b4ab0f4 not found: ID does not exist"
Sep 16 13:54:24.631853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.631819    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"]
Sep 16 13:54:24.635134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:24.635116    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-b-cf2b5d9d-kserve-7b5d4d44bd-r9dgv"]
Sep 16 13:54:25.726069 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:25.726018    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:26.059591 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:26.059515    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5c8c5a-8da8-44de-887c-5dd8af358572" path="/var/lib/kubelet/pods/cf5c8c5a-8da8-44de-887c-5dd8af358572/volumes"
Sep 16 13:54:27.282777 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.282737    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq" podUID="7bdd01d3-418b-4928-91b4-53d02760514a" containerName="main" containerID="cri-o://84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847" gracePeriod=45
Sep 16 13:54:27.555518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.555487    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:54:27.637369 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.637336    2556 generic.go:356] "Generic (PLEG): container finished" podID="7bdd01d3-418b-4928-91b4-53d02760514a" containerID="84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847" exitCode=0
Sep 16 13:54:27.637563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.637387    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"
Sep 16 13:54:27.637563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.637432    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq" event={"ID":"7bdd01d3-418b-4928-91b4-53d02760514a","Type":"ContainerDied","Data":"84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847"}
Sep 16 13:54:27.637563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.637487    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq" event={"ID":"7bdd01d3-418b-4928-91b4-53d02760514a","Type":"ContainerDied","Data":"946200a18c784370df2defef2b22cd4f37edb513e5c1b1c4b628827dfbdb3d0e"}
Sep 16 13:54:27.637563 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.637507    2556 scope.go:117] "RemoveContainer" containerID="84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847"
Sep 16 13:54:27.647600 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.647576    2556 scope.go:117] "RemoveContainer" containerID="84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847"
Sep 16 13:54:27.647891 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:54:27.647872    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847\": container with ID starting with 84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847 not found: ID does not exist" containerID="84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847"
Sep 16 13:54:27.647983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.647902    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847"} err="failed to get container status \"84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847\": rpc error: code = NotFound desc = could not find container \"84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847\": container with ID starting with 84e10982c99dadd3f2d5393c68115afa9ec2bd0d8b8f3eb74c75ebe9448d2847 not found: ID does not exist"
Sep 16 13:54:27.750057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750014    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdd01d3-418b-4928-91b4-53d02760514a-tls-certs\") pod \"7bdd01d3-418b-4928-91b4-53d02760514a\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") "
Sep 16 13:54:27.750321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750140    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zxz4\" (UniqueName: \"kubernetes.io/projected/7bdd01d3-418b-4928-91b4-53d02760514a-kube-api-access-8zxz4\") pod \"7bdd01d3-418b-4928-91b4-53d02760514a\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") "
Sep 16 13:54:27.750321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-tmp-dir\") pod \"7bdd01d3-418b-4928-91b4-53d02760514a\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") "
Sep 16 13:54:27.750321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-dshm\") pod \"7bdd01d3-418b-4928-91b4-53d02760514a\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") "
Sep 16 13:54:27.750321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750260    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-home\") pod \"7bdd01d3-418b-4928-91b4-53d02760514a\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") "
Sep 16 13:54:27.750321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750295    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-model-cache\") pod \"7bdd01d3-418b-4928-91b4-53d02760514a\" (UID: \"7bdd01d3-418b-4928-91b4-53d02760514a\") "
Sep 16 13:54:27.750606 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750499    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-home" (OuterVolumeSpecName: "home") pod "7bdd01d3-418b-4928-91b4-53d02760514a" (UID: "7bdd01d3-418b-4928-91b4-53d02760514a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:27.750675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750654    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bdd01d3-418b-4928-91b4-53d02760514a" (UID: "7bdd01d3-418b-4928-91b4-53d02760514a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:27.750675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750668    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:27.750779 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.750717    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bdd01d3-418b-4928-91b4-53d02760514a" (UID: "7bdd01d3-418b-4928-91b4-53d02760514a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:27.752506 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.752482    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-dshm" (OuterVolumeSpecName: "dshm") pod "7bdd01d3-418b-4928-91b4-53d02760514a" (UID: "7bdd01d3-418b-4928-91b4-53d02760514a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:54:27.752506 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.752501    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdd01d3-418b-4928-91b4-53d02760514a-kube-api-access-8zxz4" (OuterVolumeSpecName: "kube-api-access-8zxz4") pod "7bdd01d3-418b-4928-91b4-53d02760514a" (UID: "7bdd01d3-418b-4928-91b4-53d02760514a"). InnerVolumeSpecName "kube-api-access-8zxz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:54:27.752655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.752506    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdd01d3-418b-4928-91b4-53d02760514a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bdd01d3-418b-4928-91b4-53d02760514a" (UID: "7bdd01d3-418b-4928-91b4-53d02760514a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:54:27.851710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.851616    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:27.851710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.851646    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:27.851710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.851654    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdd01d3-418b-4928-91b4-53d02760514a-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:27.851710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.851665    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdd01d3-418b-4928-91b4-53d02760514a-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:27.851710 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.851673    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zxz4\" (UniqueName: \"kubernetes.io/projected/7bdd01d3-418b-4928-91b4-53d02760514a-kube-api-access-8zxz4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:54:27.960196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.960163    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"]
Sep 16 13:54:27.965509 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:27.965484    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/e2e-conflict-a-cf2b5d9d-kserve-7d557c9fb6-w9dqq"]
Sep 16 13:54:28.059904 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:28.059868    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bdd01d3-418b-4928-91b4-53d02760514a" path="/var/lib/kubelet/pods/7bdd01d3-418b-4928-91b4-53d02760514a/volumes"
Sep 16 13:54:50.314006 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:50.313964    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:51.068162 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:51.068129    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:51.785981 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:51.785950    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:52.506738 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:52.506704    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:53.182179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:53.182144    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:53.987204 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:53.987168    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:54.775022 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:54.774977    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:55.440922 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:55.440886    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:56.401611 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:56.401569    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:54:58.836370 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:54:58.836333    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:55:05.655692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:05.655659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:05.719468 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:05.719417    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:05.783117 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:05.783068    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:05.872529 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:05.872499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:05.927034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:05.927003    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:06.262151 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:06.262032    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:07.100914 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:07.100863    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:07.779683 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:07.779649    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:08.482004 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:08.481971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:09.134951 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:09.134903    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:09.235490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:09.235453    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:10.251714 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:10.251679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:10.679351 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:10.679321    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:11.845528 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.845498    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"]
Sep 16 13:55:11.845971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.845958    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5c8c5a-8da8-44de-887c-5dd8af358572" containerName="main"
Sep 16 13:55:11.845971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.845972    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5c8c5a-8da8-44de-887c-5dd8af358572" containerName="main"
Sep 16 13:55:11.846057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.845983    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081686fa-4b02-41cc-af6b-4416ba0713bf" containerName="main"
Sep 16 13:55:11.846057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.845989    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="081686fa-4b02-41cc-af6b-4416ba0713bf" containerName="main"
Sep 16 13:55:11.846057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.846008    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdd01d3-418b-4928-91b4-53d02760514a" containerName="main"
Sep 16 13:55:11.846057 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.846015    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdd01d3-418b-4928-91b4-53d02760514a" containerName="main"
Sep 16 13:55:11.846205 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.846101    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="081686fa-4b02-41cc-af6b-4416ba0713bf" containerName="main"
Sep 16 13:55:11.846205 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.846112    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5c8c5a-8da8-44de-887c-5dd8af358572" containerName="main"
Sep 16 13:55:11.846205 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.846120    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdd01d3-418b-4928-91b4-53d02760514a" containerName="main"
Sep 16 13:55:11.850559 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.850536    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:11.852968 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.852944    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc8c65\"/\"distinct-w4-fc8c65-kserve-self-signed-certs\""
Sep 16 13:55:11.857954 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.857931    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"]
Sep 16 13:55:11.867625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.867597    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:11.901857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.901818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41cfe8c-342a-4309-b452-fd2e8ca95946-tls-certs\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:11.902047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.901865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-home\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:11.902047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.901938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-model-cache\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:11.902047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.901959    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-dshm\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:11.902047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.902010    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-tmp-dir\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:11.902047 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:11.902046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lnbd\" (UniqueName: \"kubernetes.io/projected/e41cfe8c-342a-4309-b452-fd2e8ca95946-kube-api-access-4lnbd\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-tmp-dir\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003750 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lnbd\" (UniqueName: \"kubernetes.io/projected/e41cfe8c-342a-4309-b452-fd2e8ca95946-kube-api-access-4lnbd\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003750 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41cfe8c-342a-4309-b452-fd2e8ca95946-tls-certs\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003750 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-home\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003810    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-model-cache\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-dshm\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.003999 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.003979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-tmp-dir\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.004124 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.004083    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-home\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.004170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.004157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-model-cache\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.006155 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.006130    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-dshm\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.006390 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.006369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41cfe8c-342a-4309-b452-fd2e8ca95946-tls-certs\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.011280 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.011260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lnbd\" (UniqueName: \"kubernetes.io/projected/e41cfe8c-342a-4309-b452-fd2e8ca95946-kube-api-access-4lnbd\") pod \"distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") " pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.161379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.161281    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:12.433902 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.433874    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"]
Sep 16 13:55:12.434789 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:55:12.434764    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode41cfe8c_342a_4309_b452_fd2e8ca95946.slice/crio-155ab3535e4224f8ef7017327ac4fa78fadf769fb9b532ed1f3fd616b43d57ff WatchSource:0}: Error finding container 155ab3535e4224f8ef7017327ac4fa78fadf769fb9b532ed1f3fd616b43d57ff: Status 404 returned error can't find the container with id 155ab3535e4224f8ef7017327ac4fa78fadf769fb9b532ed1f3fd616b43d57ff
Sep 16 13:55:12.931503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.931465    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz" event={"ID":"e41cfe8c-342a-4309-b452-fd2e8ca95946","Type":"ContainerStarted","Data":"bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4"}
Sep 16 13:55:12.931503 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.931509    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz" event={"ID":"e41cfe8c-342a-4309-b452-fd2e8ca95946","Type":"ContainerStarted","Data":"155ab3535e4224f8ef7017327ac4fa78fadf769fb9b532ed1f3fd616b43d57ff"}
Sep 16 13:55:12.949275 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:12.949226    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz" podStartSLOduration=1.9492125489999998 podStartE2EDuration="1.949212549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:12.946203915 +0000 UTC m=+1323.412140834" watchObservedRunningTime="2026-09-16 13:55:12.949212549 +0000 UTC m=+1323.415149458"
Sep 16 13:55:14.114714 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:14.114675    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"]
Sep 16 13:55:14.208886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:14.208855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:19.138072 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:19.138038    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:22.161746 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:22.161694    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:22.161746 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:22.161755    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:22.174615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:22.174593    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:23.006795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:23.006759    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:55:26.266447 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:26.266410    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:29.136016 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.135976    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s" podUID="cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" containerName="main" containerID="cri-o://6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6" gracePeriod=45
Sep 16 13:55:29.400325 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.400270    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:55:29.497513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497480    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-dshm\") pod \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") "
Sep 16 13:55:29.497701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497551    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tmp-dir\") pod \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") "
Sep 16 13:55:29.497701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497602    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tls-certs\") pod \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") "
Sep 16 13:55:29.497701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497625    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76282\" (UniqueName: \"kubernetes.io/projected/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-kube-api-access-76282\") pod \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") "
Sep 16 13:55:29.497701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497693    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-model-cache\") pod \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") "
Sep 16 13:55:29.497933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497799    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-home\") pod \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\" (UID: \"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd\") "
Sep 16 13:55:29.497933 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.497858    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" (UID: "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:55:29.498141 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.498113    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-model-cache" (OuterVolumeSpecName: "model-cache") pod "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" (UID: "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:55:29.498246 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.498221    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-home" (OuterVolumeSpecName: "home") pod "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" (UID: "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:55:29.498530 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.498512    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:55:29.498588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.498537    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:55:29.498588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.498552    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:55:29.500046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.500025    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-dshm" (OuterVolumeSpecName: "dshm") pod "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" (UID: "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:55:29.500046 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.500033    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-kube-api-access-76282" (OuterVolumeSpecName: "kube-api-access-76282") pod "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" (UID: "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd"). InnerVolumeSpecName "kube-api-access-76282". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:55:29.500228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.500047    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" (UID: "cde9bb33-7d27-4f67-9d3b-3643e2a56ffd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:55:29.600174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.600136    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:55:29.600174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.600166    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:55:29.600174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.600180    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76282\" (UniqueName: \"kubernetes.io/projected/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd-kube-api-access-76282\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:55:29.639971 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:29.639933    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:55:30.039575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.039538    2556 generic.go:356] "Generic (PLEG): container finished" podID="cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" containerID="6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6" exitCode=0
Sep 16 13:55:30.039776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.039593    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"
Sep 16 13:55:30.039776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.039646    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s" event={"ID":"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd","Type":"ContainerDied","Data":"6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6"}
Sep 16 13:55:30.039776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.039685    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s" event={"ID":"cde9bb33-7d27-4f67-9d3b-3643e2a56ffd","Type":"ContainerDied","Data":"d29f964709f6fa27c73dee1c4768222f8108105540e88d0044fd04f147bed7fb"}
Sep 16 13:55:30.039776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.039701    2556 scope.go:117] "RemoveContainer" containerID="6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6"
Sep 16 13:55:30.049484 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.049468    2556 scope.go:117] "RemoveContainer" containerID="6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6"
Sep 16 13:55:30.049731 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:55:30.049714    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6\": container with ID starting with 6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6 not found: ID does not exist" containerID="6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6"
Sep 16 13:55:30.049787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.049737    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6"} err="failed to get container status \"6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6\": rpc error: code = NotFound desc = could not find container \"6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6\": container with ID starting with 6d49e2a6fb4c428f2afa6fc9eb771a687314b90a496d987905895eb7c1cfade6 not found: ID does not exist"
Sep 16 13:55:30.064247 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.064219    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"]
Sep 16 13:55:30.067038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:30.067014    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3be042d3/e2e-lifecycle-model-3be042d3-kserve-55cdbc859f-wck2s"]
Sep 16 13:55:32.059429 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:32.059345    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" path="/var/lib/kubelet/pods/cde9bb33-7d27-4f67-9d3b-3643e2a56ffd/volumes"
Sep 16 13:55:50.256340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:50.256301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:51.048976 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:51.048943    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:51.432769 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:51.432729    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:55:56.663889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:56.663845    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:57.230883 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.230847    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"]
Sep 16 13:55:57.231514 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.231492    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" containerName="main"
Sep 16 13:55:57.231613 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.231518    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" containerName="main"
Sep 16 13:55:57.232070 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.231667    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde9bb33-7d27-4f67-9d3b-3643e2a56ffd" containerName="main"
Sep 16 13:55:57.235208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.235188    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.237236 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.237216    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7d96c7\"/\"embedding-w0-7d96c7-kserve-self-signed-certs\""
Sep 16 13:55:57.242632 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.242610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"]
Sep 16 13:55:57.259562 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.259536    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:57.289667 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.289627    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-home\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.289860 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.289699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tmp-dir\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.289860 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.289742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dt7n\" (UniqueName: \"kubernetes.io/projected/b5a62d20-23ca-4388-86c1-e6d8247c83ea-kube-api-access-9dt7n\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.289860 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.289835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-dshm\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.290028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.289961    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-model-cache\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.290028 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.289997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tls-certs\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391240 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391205    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tmp-dir\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391240 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391247    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dt7n\" (UniqueName: \"kubernetes.io/projected/b5a62d20-23ca-4388-86c1-e6d8247c83ea-kube-api-access-9dt7n\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-dshm\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-model-cache\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tls-certs\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391671 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-home\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.391945 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391920    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tmp-dir\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.392035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391969    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-model-cache\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.392035 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.391992    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-home\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.394498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.394475    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-dshm\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.394731 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.394709    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tls-certs\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.398807 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.398784    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dt7n\" (UniqueName: \"kubernetes.io/projected/b5a62d20-23ca-4388-86c1-e6d8247c83ea-kube-api-access-9dt7n\") pod \"embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") " pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.546349 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.546253    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:55:57.776771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.776741    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:55:57.818408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:57.818379    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"]
Sep 16 13:55:57.819616 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:55:57.819584    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a62d20_23ca_4388_86c1_e6d8247c83ea.slice/crio-a291badea44c2b4c7863e7fbf78f2055ccd740eb4f933b36d596db4248bea7e4 WatchSource:0}: Error finding container a291badea44c2b4c7863e7fbf78f2055ccd740eb4f933b36d596db4248bea7e4: Status 404 returned error can't find the container with id a291badea44c2b4c7863e7fbf78f2055ccd740eb4f933b36d596db4248bea7e4
Sep 16 13:55:58.230994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:58.230964    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn" event={"ID":"b5a62d20-23ca-4388-86c1-e6d8247c83ea","Type":"ContainerStarted","Data":"5932aeb6cf5d8d655f287cbf1afbef83ca96f8e39fac0aded52a7d1719f6e32c"}
Sep 16 13:55:58.231198 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:58.231005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn" event={"ID":"b5a62d20-23ca-4388-86c1-e6d8247c83ea","Type":"ContainerStarted","Data":"a291badea44c2b4c7863e7fbf78f2055ccd740eb4f933b36d596db4248bea7e4"}
Sep 16 13:55:58.247017 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:55:58.246973    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn" podStartSLOduration=1.246960552 podStartE2EDuration="1.246960552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:55:58.245625895 +0000 UTC m=+1368.711562835" watchObservedRunningTime="2026-09-16 13:55:58.246960552 +0000 UTC m=+1368.712897458"
Sep 16 13:56:01.745833 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:01.745799    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:03.753787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:03.753754    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:07.546577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:07.546540    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:56:07.547001 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:07.546631    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:56:07.559199 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:07.559176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:56:08.305328 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:08.305299    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:56:11.011898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:11.011853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:11.111998 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:11.111969    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:11.249378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:11.249348    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"]
Sep 16 13:56:11.249620 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:11.249593    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy" containerID="cri-o://28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf" gracePeriod=30
Sep 16 13:56:11.269384 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:11.269295    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:12.540273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:12.540236    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:12.636826 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:12.636776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:12.738987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:12.738958    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:13.363387 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:13.363351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:13.468233 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:13.468196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:13.563577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:13.563542    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:14.843548 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:14.843512    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:16.512588 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.512525    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:56:16.593361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593325    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-kube-api-access-pstlf\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.593568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593378    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-certs\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.593568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593452    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-podinfo\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.593568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593476    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-socket\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.593568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593501    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-token\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.593568 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593537    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istiod-ca-cert\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.593835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593673    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:16.593835 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593808    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:16.594026 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.593994    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:56:16.594026 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594014    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-envoy\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.594403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594051    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-credential-socket\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.594403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594144    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-data\") pod \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\" (UID: \"af3762dc-7d18-4eba-ad8d-3f06dc1dba62\") "
Sep 16 13:56:16.594523 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594480    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:16.594583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594519    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-data" (OuterVolumeSpecName: "istio-data") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:16.594583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594576    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.594703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594589    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.594703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594599    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.594703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.594611    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.595931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.595908    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-kube-api-access-pstlf" (OuterVolumeSpecName: "kube-api-access-pstlf") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "kube-api-access-pstlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:56:16.596367 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.596335    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:56:16.596367 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.596346    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-token" (OuterVolumeSpecName: "istio-token") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:56:16.596492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.596400    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af3762dc-7d18-4eba-ad8d-3f06dc1dba62" (UID: "af3762dc-7d18-4eba-ad8d-3f06dc1dba62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:56:16.695772 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.695736    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.695772 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.695766    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.695772 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.695777    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstlf\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-kube-api-access-pstlf\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.696014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.695786    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:16.696014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:16.695794    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af3762dc-7d18-4eba-ad8d-3f06dc1dba62-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:56:17.350957 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.350927    2556 generic.go:356] "Generic (PLEG): container finished" podID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerID="28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf" exitCode=0
Sep 16 13:56:17.351195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.350981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" event={"ID":"af3762dc-7d18-4eba-ad8d-3f06dc1dba62","Type":"ContainerDied","Data":"28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf"}
Sep 16 13:56:17.351195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.350991    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"
Sep 16 13:56:17.351195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.351022    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n" event={"ID":"af3762dc-7d18-4eba-ad8d-3f06dc1dba62","Type":"ContainerDied","Data":"7c435d12b5ee6d046a51e7f3b73c046bc55251a4b0ce161f143151b51bc63879"}
Sep 16 13:56:17.351195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.351045    2556 scope.go:117] "RemoveContainer" containerID="28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf"
Sep 16 13:56:17.361476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.361453    2556 scope.go:117] "RemoveContainer" containerID="81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"
Sep 16 13:56:17.371615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.371535    2556 scope.go:117] "RemoveContainer" containerID="28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf"
Sep 16 13:56:17.373757 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:56:17.373725    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf\": container with ID starting with 28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf not found: ID does not exist" containerID="28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf"
Sep 16 13:56:17.373881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.373763    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf"} err="failed to get container status \"28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf\": rpc error: code = NotFound desc = could not find container \"28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf\": container with ID starting with 28ed67cc9ad1f0d0cad3d2769d9c1945162794e72cfa498482c53cf55ac71abf not found: ID does not exist"
Sep 16 13:56:17.373881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.373785    2556 scope.go:117] "RemoveContainer" containerID="81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"
Sep 16 13:56:17.374216 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:56:17.374182    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec\": container with ID starting with 81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec not found: ID does not exist" containerID="81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"
Sep 16 13:56:17.374331 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.374214    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec"} err="failed to get container status \"81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec\": rpc error: code = NotFound desc = could not find container \"81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec\": container with ID starting with 81e282003890453e1f985ea6cf0d6497f010d8b2dd7dbb7cb9e3b7f7e595c2ec not found: ID does not exist"
Sep 16 13:56:17.374435 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.374412    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"]
Sep 16 13:56:17.377913 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:17.377892    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3be042d3-openshift-default-6495657988-9cj8n"]
Sep 16 13:56:18.060446 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:18.060416    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" path="/var/lib/kubelet/pods/af3762dc-7d18-4eba-ad8d-3f06dc1dba62/volumes"
Sep 16 13:56:20.625675 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:20.625647    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:20.688351 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:20.688317    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:21.320584 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:21.320553    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:26.956001 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:26.955964    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:27.017159 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:27.017125    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:27.082429 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:27.082398    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:27.182579 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:27.182544    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:29.839852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:29.839801    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:56:37.670134 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:37.670073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:37.895714 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:37.895676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:39.547276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:39.547235    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:58.476478 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:58.476434    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:58.854414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:58.854321    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:58.957223 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:58.957184    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:56:59.785742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:56:59.785709    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:00.485200 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:00.485167    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:01.247613 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:01.247577    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:02.172666 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:02.172626    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:02.857600 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:02.857569    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:03.468245 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:03.467325    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"]
Sep 16 13:57:03.479190 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:03.479154    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"]
Sep 16 13:57:03.500643 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:03.500607    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"]
Sep 16 13:57:03.547436 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:03.547408    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:03.680463 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:03.680418    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:04.638974 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:04.638940    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:04.999420 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:04.999366    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:05.100195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.100155    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:05.369811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.369705    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:05.733025 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.732981    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"]
Sep 16 13:57:05.733542 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.733526    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy"
Sep 16 13:57:05.733542 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.733543    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy"
Sep 16 13:57:05.733662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.733552    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy"
Sep 16 13:57:05.733662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.733557    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy"
Sep 16 13:57:05.733662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.733627    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy"
Sep 16 13:57:05.737113 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.737075    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.739056 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.739041    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-44f000-openshift-default-dockercfg-2vlmd\""
Sep 16 13:57:05.751452 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.751423    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"]
Sep 16 13:57:05.772051 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.772007    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:05.811021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.810950    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-credential-socket\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istiod-ca-cert\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811474 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-envoy\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811474 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811414    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhn7d\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-kube-api-access-hhn7d\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811602    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-socket\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811693 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-data\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811769 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811749    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-podinfo\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811823 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-certs\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.811872 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.811833    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-token\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913123 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-credential-socket\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913141    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istiod-ca-cert\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913182    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-envoy\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913206    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhn7d\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-kube-api-access-hhn7d\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-socket\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-data\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-podinfo\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-certs\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913690 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913497    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-token\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913834 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913776    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-data\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.913834 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-socket\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.914038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913853    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-certs\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.914038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.913982    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-credential-socket\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.914363 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.914341    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istiod-ca-cert\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.915616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.915591    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-envoy\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.916015 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.915996    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-podinfo\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.920695 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.920666    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-token\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:05.920695 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:05.920685    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhn7d\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-kube-api-access-hhn7d\") pod \"e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") " pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:06.048272 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.048174    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:06.332647 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:57:06.332617    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6920de26_b04b_4bc1_a5da_f4a20be5ee23.slice/crio-cf01dd814753979072b3debd86293c2fb56205c262560734fbbcb136c48d9972 WatchSource:0}: Error finding container cf01dd814753979072b3debd86293c2fb56205c262560734fbbcb136c48d9972: Status 404 returned error can't find the container with id cf01dd814753979072b3debd86293c2fb56205c262560734fbbcb136c48d9972
Sep 16 13:57:06.333555 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.333521    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"]
Sep 16 13:57:06.335177 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.335142    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:57:06.335270 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.335211    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:57:06.335270 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.335241    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:57:06.439843 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.439802    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:57:06.621393 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.621296    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:06.673292 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.673251    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" event={"ID":"6920de26-b04b-4bc1-a5da-f4a20be5ee23","Type":"ContainerStarted","Data":"6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"}
Sep 16 13:57:06.673519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.673313    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" event={"ID":"6920de26-b04b-4bc1-a5da-f4a20be5ee23","Type":"ContainerStarted","Data":"cf01dd814753979072b3debd86293c2fb56205c262560734fbbcb136c48d9972"}
Sep 16 13:57:06.691540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:06.691477    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podStartSLOduration=1.691455004 podStartE2EDuration="1.691455004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:06.691005973 +0000 UTC m=+1437.156942910" watchObservedRunningTime="2026-09-16 13:57:06.691455004 +0000 UTC m=+1437.157391913"
Sep 16 13:57:07.049309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:07.049273    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:07.054660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:07.054629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:07.679286 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:07.679253    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:07.680361 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:07.680335    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:57:07.982469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:07.982377    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:08.688400 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:08.688364    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:08.778326 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:08.778290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:09.061962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:09.061871    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:09.913854 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:09.913823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:11.015722 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:11.015691    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:11.674273 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:11.674237    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:12.489808 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:12.489760    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:13.294565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:13.294507    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:13.395298 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:13.395263    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:13.523810 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:13.523773    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:13.740289 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:13.740260    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:13.839295 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:13.839253    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:14.293269 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:14.293224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:15.072328 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:15.072296    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:15.782127 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:15.782057    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.137695 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.137659    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"]
Sep 16 13:57:18.138216 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.137940    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" podUID="ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" containerName="maas-api" containerID="cri-o://75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c" gracePeriod=30
Sep 16 13:57:18.170662 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.170617    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.281067 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.281023    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.359491 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.359454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.412513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.412483    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:57:18.425170 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.425136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.464903 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.464868    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-metrics-certs\") pod \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") "
Sep 16 13:57:18.465153 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.464967    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mwg7\" (UniqueName: \"kubernetes.io/projected/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-kube-api-access-7mwg7\") pod \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") "
Sep 16 13:57:18.465153 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.465012    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls\") pod \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") "
Sep 16 13:57:18.465929 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.465868    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" (UID: "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:57:18.466154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.466131    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-openshift-service-ca\") pod \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\" (UID: \"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a\") "
Sep 16 13:57:18.466850 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.466823    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.468179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.468149    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-kube-api-access-7mwg7" (OuterVolumeSpecName: "kube-api-access-7mwg7") pod "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" (UID: "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a"). InnerVolumeSpecName "kube-api-access-7mwg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:18.468425 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.468222    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" (UID: "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:18.468887 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.468866    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" (UID: "ab5c28d4-bc34-47b7-9e3a-14620fd6f16a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:18.495517 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.495238    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2" podUID="ae901fb1-999e-4291-999d-bdfa6d479ded" containerName="main" containerID="cri-o://98441fb39a0cb6f6ec85163c8e42696b60835a6170f1f7f85a84f4c6b3e26d19" gracePeriod=45
Sep 16 13:57:18.501954 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.501772    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn" podUID="b5a62d20-23ca-4388-86c1-e6d8247c83ea" containerName="main" containerID="cri-o://5932aeb6cf5d8d655f287cbf1afbef83ca96f8e39fac0aded52a7d1719f6e32c" gracePeriod=45
Sep 16 13:57:18.522354 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.522309    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67" podUID="85386041-679a-4b3a-a873-6b1299a43674" containerName="main" containerID="cri-o://21c3b6585d1fea95e3eb98bc6b7008e706c08ade80aa2a97eb263b6e7cc34e8f" gracePeriod=45
Sep 16 13:57:18.568513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.568467    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mwg7\" (UniqueName: \"kubernetes.io/projected/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-kube-api-access-7mwg7\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.568513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.568509    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.568513 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.568524    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.706918 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.706877    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:18.752712 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.752676    2556 generic.go:356] "Generic (PLEG): container finished" podID="ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" containerID="75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c" exitCode=0
Sep 16 13:57:18.752881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.752724    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"
Sep 16 13:57:18.752881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.752769    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" event={"ID":"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a","Type":"ContainerDied","Data":"75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c"}
Sep 16 13:57:18.752881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.752840    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f" event={"ID":"ab5c28d4-bc34-47b7-9e3a-14620fd6f16a","Type":"ContainerDied","Data":"eb1adb92452be4cdaec7e348c93079c8f0365caa7d8718815ba5f5ed8acb255d"}
Sep 16 13:57:18.752881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.752876    2556 scope.go:117] "RemoveContainer" containerID="75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c"
Sep 16 13:57:18.755053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.755024    2556 generic.go:356] "Generic (PLEG): container finished" podID="ae901fb1-999e-4291-999d-bdfa6d479ded" containerID="98441fb39a0cb6f6ec85163c8e42696b60835a6170f1f7f85a84f4c6b3e26d19" exitCode=0
Sep 16 13:57:18.755296 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.755220    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2" event={"ID":"ae901fb1-999e-4291-999d-bdfa6d479ded","Type":"ContainerDied","Data":"98441fb39a0cb6f6ec85163c8e42696b60835a6170f1f7f85a84f4c6b3e26d19"}
Sep 16 13:57:18.757637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.757488    2556 generic.go:356] "Generic (PLEG): container finished" podID="b5a62d20-23ca-4388-86c1-e6d8247c83ea" containerID="5932aeb6cf5d8d655f287cbf1afbef83ca96f8e39fac0aded52a7d1719f6e32c" exitCode=0
Sep 16 13:57:18.757637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.757580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn" event={"ID":"b5a62d20-23ca-4388-86c1-e6d8247c83ea","Type":"ContainerDied","Data":"5932aeb6cf5d8d655f287cbf1afbef83ca96f8e39fac0aded52a7d1719f6e32c"}
Sep 16 13:57:18.759713 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.759692    2556 generic.go:356] "Generic (PLEG): container finished" podID="85386041-679a-4b3a-a873-6b1299a43674" containerID="21c3b6585d1fea95e3eb98bc6b7008e706c08ade80aa2a97eb263b6e7cc34e8f" exitCode=0
Sep 16 13:57:18.759841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.759741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67" event={"ID":"85386041-679a-4b3a-a873-6b1299a43674","Type":"ContainerDied","Data":"21c3b6585d1fea95e3eb98bc6b7008e706c08ade80aa2a97eb263b6e7cc34e8f"}
Sep 16 13:57:18.769962 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.769918    2556 scope.go:117] "RemoveContainer" containerID="75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c"
Sep 16 13:57:18.770364 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:57:18.770330    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c\": container with ID starting with 75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c not found: ID does not exist" containerID="75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c"
Sep 16 13:57:18.770466 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.770377    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c"} err="failed to get container status \"75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c\": rpc error: code = NotFound desc = could not find container \"75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c\": container with ID starting with 75524ca4c7661f1c7d049242b76f851d5b1f65c71f15c713af2eda829a1d1a6c not found: ID does not exist"
Sep 16 13:57:18.780589 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.780532    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"]
Sep 16 13:57:18.784786 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.784751    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7d96c7-58bf5949cc-5sz2f"]
Sep 16 13:57:18.806841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.806732    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:57:18.832021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.831993    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:57:18.847178 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.847143    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:57:18.872852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.872824    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-tmp-dir\") pod \"ae901fb1-999e-4291-999d-bdfa6d479ded\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") "
Sep 16 13:57:18.872852 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.872857    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-dshm\") pod \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") "
Sep 16 13:57:18.873152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.872874    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-dshm\") pod \"ae901fb1-999e-4291-999d-bdfa6d479ded\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") "
Sep 16 13:57:18.873152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.872908    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tmp-dir\") pod \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") "
Sep 16 13:57:18.873152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.872953    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-home\") pod \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") "
Sep 16 13:57:18.873152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.872999    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-model-cache\") pod \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") "
Sep 16 13:57:18.873152 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873130    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-home\") pod \"85386041-679a-4b3a-a873-6b1299a43674\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873169    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tls-certs\") pod \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873198    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-model-cache\") pod \"ae901fb1-999e-4291-999d-bdfa6d479ded\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873232    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-model-cache\") pod \"85386041-679a-4b3a-a873-6b1299a43674\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873251    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae901fb1-999e-4291-999d-bdfa6d479ded" (UID: "ae901fb1-999e-4291-999d-bdfa6d479ded"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873306    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae901fb1-999e-4291-999d-bdfa6d479ded-tls-certs\") pod \"ae901fb1-999e-4291-999d-bdfa6d479ded\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873376    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-dshm\") pod \"85386041-679a-4b3a-a873-6b1299a43674\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873401    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-tmp-dir\") pod \"85386041-679a-4b3a-a873-6b1299a43674\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") "
Sep 16 13:57:18.873455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873437    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dt7n\" (UniqueName: \"kubernetes.io/projected/b5a62d20-23ca-4388-86c1-e6d8247c83ea-kube-api-access-9dt7n\") pod \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\" (UID: \"b5a62d20-23ca-4388-86c1-e6d8247c83ea\") "
Sep 16 13:57:18.873839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873463    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5a62d20-23ca-4388-86c1-e6d8247c83ea" (UID: "b5a62d20-23ca-4388-86c1-e6d8247c83ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.873839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873475    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-home\") pod \"ae901fb1-999e-4291-999d-bdfa6d479ded\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") "
Sep 16 13:57:18.873839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873528    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85386041-679a-4b3a-a873-6b1299a43674-tls-certs\") pod \"85386041-679a-4b3a-a873-6b1299a43674\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") "
Sep 16 13:57:18.873839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873557    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58swp\" (UniqueName: \"kubernetes.io/projected/85386041-679a-4b3a-a873-6b1299a43674-kube-api-access-58swp\") pod \"85386041-679a-4b3a-a873-6b1299a43674\" (UID: \"85386041-679a-4b3a-a873-6b1299a43674\") "
Sep 16 13:57:18.873839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873596    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mntz\" (UniqueName: \"kubernetes.io/projected/ae901fb1-999e-4291-999d-bdfa6d479ded-kube-api-access-7mntz\") pod \"ae901fb1-999e-4291-999d-bdfa6d479ded\" (UID: \"ae901fb1-999e-4291-999d-bdfa6d479ded\") "
Sep 16 13:57:18.873839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873663    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-home" (OuterVolumeSpecName: "home") pod "ae901fb1-999e-4291-999d-bdfa6d479ded" (UID: "ae901fb1-999e-4291-999d-bdfa6d479ded"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.874165 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.873846    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-home" (OuterVolumeSpecName: "home") pod "85386041-679a-4b3a-a873-6b1299a43674" (UID: "85386041-679a-4b3a-a873-6b1299a43674"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.874165 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874140    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.874165 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874160    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.874309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874174    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.874309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874185    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.874309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874197    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae901fb1-999e-4291-999d-bdfa6d479ded" (UID: "ae901fb1-999e-4291-999d-bdfa6d479ded"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.874538 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874514    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85386041-679a-4b3a-a873-6b1299a43674" (UID: "85386041-679a-4b3a-a873-6b1299a43674"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.874650 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.874609    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-home" (OuterVolumeSpecName: "home") pod "b5a62d20-23ca-4388-86c1-e6d8247c83ea" (UID: "b5a62d20-23ca-4388-86c1-e6d8247c83ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.878869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.877517    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-model-cache" (OuterVolumeSpecName: "model-cache") pod "85386041-679a-4b3a-a873-6b1299a43674" (UID: "85386041-679a-4b3a-a873-6b1299a43674"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.878869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.877516    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5a62d20-23ca-4388-86c1-e6d8247c83ea" (UID: "b5a62d20-23ca-4388-86c1-e6d8247c83ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.878869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.878270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a62d20-23ca-4388-86c1-e6d8247c83ea-kube-api-access-9dt7n" (OuterVolumeSpecName: "kube-api-access-9dt7n") pod "b5a62d20-23ca-4388-86c1-e6d8247c83ea" (UID: "b5a62d20-23ca-4388-86c1-e6d8247c83ea"). InnerVolumeSpecName "kube-api-access-9dt7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:18.878869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.878513    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae901fb1-999e-4291-999d-bdfa6d479ded-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae901fb1-999e-4291-999d-bdfa6d479ded" (UID: "ae901fb1-999e-4291-999d-bdfa6d479ded"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:18.878869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.878858    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-dshm" (OuterVolumeSpecName: "dshm") pod "ae901fb1-999e-4291-999d-bdfa6d479ded" (UID: "ae901fb1-999e-4291-999d-bdfa6d479ded"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.879459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.879426    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85386041-679a-4b3a-a873-6b1299a43674-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85386041-679a-4b3a-a873-6b1299a43674" (UID: "85386041-679a-4b3a-a873-6b1299a43674"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:18.879999 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.879961    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-dshm" (OuterVolumeSpecName: "dshm") pod "b5a62d20-23ca-4388-86c1-e6d8247c83ea" (UID: "b5a62d20-23ca-4388-86c1-e6d8247c83ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.880319 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.880287    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5a62d20-23ca-4388-86c1-e6d8247c83ea" (UID: "b5a62d20-23ca-4388-86c1-e6d8247c83ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:57:18.880319 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.880265    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-dshm" (OuterVolumeSpecName: "dshm") pod "85386041-679a-4b3a-a873-6b1299a43674" (UID: "85386041-679a-4b3a-a873-6b1299a43674"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:18.880824 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.880792    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae901fb1-999e-4291-999d-bdfa6d479ded-kube-api-access-7mntz" (OuterVolumeSpecName: "kube-api-access-7mntz") pod "ae901fb1-999e-4291-999d-bdfa6d479ded" (UID: "ae901fb1-999e-4291-999d-bdfa6d479ded"). InnerVolumeSpecName "kube-api-access-7mntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:18.880917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.880886    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85386041-679a-4b3a-a873-6b1299a43674-kube-api-access-58swp" (OuterVolumeSpecName: "kube-api-access-58swp") pod "85386041-679a-4b3a-a873-6b1299a43674" (UID: "85386041-679a-4b3a-a873-6b1299a43674"). InnerVolumeSpecName "kube-api-access-58swp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975674    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975705    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975715    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975722    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae901fb1-999e-4291-999d-bdfa6d479ded-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975729    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975737    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85386041-679a-4b3a-a873-6b1299a43674-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975745    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dt7n\" (UniqueName: \"kubernetes.io/projected/b5a62d20-23ca-4388-86c1-e6d8247c83ea-kube-api-access-9dt7n\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975754    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85386041-679a-4b3a-a873-6b1299a43674-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975764    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58swp\" (UniqueName: \"kubernetes.io/projected/85386041-679a-4b3a-a873-6b1299a43674-kube-api-access-58swp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.975767 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975771    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mntz\" (UniqueName: \"kubernetes.io/projected/ae901fb1-999e-4291-999d-bdfa6d479ded-kube-api-access-7mntz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.976239 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975780    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.976239 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975789    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae901fb1-999e-4291-999d-bdfa6d479ded-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.976239 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975796    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:18.976239 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:18.975804    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5a62d20-23ca-4388-86c1-e6d8247c83ea-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:19.558994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.558962    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:19.672419 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.672379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:19.767692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.767666    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"
Sep 16 13:57:19.767886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.767708    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2" event={"ID":"ae901fb1-999e-4291-999d-bdfa6d479ded","Type":"ContainerDied","Data":"d964a9f7b28c00d144cb578d880717033ccad91307439a6c44ef338097968739"}
Sep 16 13:57:19.767886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.767755    2556 scope.go:117] "RemoveContainer" containerID="98441fb39a0cb6f6ec85163c8e42696b60835a6170f1f7f85a84f4c6b3e26d19"
Sep 16 13:57:19.769590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.769564    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn" event={"ID":"b5a62d20-23ca-4388-86c1-e6d8247c83ea","Type":"ContainerDied","Data":"a291badea44c2b4c7863e7fbf78f2055ccd740eb4f933b36d596db4248bea7e4"}
Sep 16 13:57:19.769590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.769572    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"
Sep 16 13:57:19.771697 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.771675    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"
Sep 16 13:57:19.771836 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.771694    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67" event={"ID":"85386041-679a-4b3a-a873-6b1299a43674","Type":"ContainerDied","Data":"83a31b6a7dcb02a1a82049e2fbcb0e38cfcc0d55934621f507dfd6c32bef3618"}
Sep 16 13:57:19.779321 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.779303    2556 scope.go:117] "RemoveContainer" containerID="5932aeb6cf5d8d655f287cbf1afbef83ca96f8e39fac0aded52a7d1719f6e32c"
Sep 16 13:57:19.789702 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.789682    2556 scope.go:117] "RemoveContainer" containerID="21c3b6585d1fea95e3eb98bc6b7008e706c08ade80aa2a97eb263b6e7cc34e8f"
Sep 16 13:57:19.795742 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.795716    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"]
Sep 16 13:57:19.801600 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.801578    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/embedding-w0-7d96c7-kserve-5669d4c97d-f7cfn"]
Sep 16 13:57:19.810445 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.810376    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"]
Sep 16 13:57:19.814466 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.814441    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/distinct-2-w0-7d96c7-kserve-75d59b5b5d-k89k2"]
Sep 16 13:57:19.823276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.823240    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"]
Sep 16 13:57:19.826741 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:19.826713    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7d96c7/simulator-w0-7d96c7-kserve-94dbb6f5-vdd67"]
Sep 16 13:57:20.060545 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:20.060458    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85386041-679a-4b3a-a873-6b1299a43674" path="/var/lib/kubelet/pods/85386041-679a-4b3a-a873-6b1299a43674/volumes"
Sep 16 13:57:20.060891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:20.060878    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" path="/var/lib/kubelet/pods/ab5c28d4-bc34-47b7-9e3a-14620fd6f16a/volumes"
Sep 16 13:57:20.061344 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:20.061330    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae901fb1-999e-4291-999d-bdfa6d479ded" path="/var/lib/kubelet/pods/ae901fb1-999e-4291-999d-bdfa6d479ded/volumes"
Sep 16 13:57:20.062001 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:20.061986    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5a62d20-23ca-4388-86c1-e6d8247c83ea" path="/var/lib/kubelet/pods/b5a62d20-23ca-4388-86c1-e6d8247c83ea/volumes"
Sep 16 13:57:20.579884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:20.579846    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:21.274590 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:21.274539    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:22.099519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:22.099486    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:23.813628 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:23.813581    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:23.914142 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:23.914069    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:24.553034 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:24.553000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:24.615967 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:24.615928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:24.795607 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:24.795572    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"]
Sep 16 13:57:24.795857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:24.795830    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy" containerID="cri-o://d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca" gracePeriod=30
Sep 16 13:57:24.804928 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:24.804807    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:30.055983 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.055948    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:57:30.096553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096465    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-certs\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096553 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096521    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-data\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096590    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb835d-2cb2-4c93-af1b-38a95406c14b-istiod-ca-cert\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096646    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-token\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096702    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-podinfo\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096930 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096789    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgj5r\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-kube-api-access-jgj5r\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096930 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096831    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:30.096930 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096849    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-data" (OuterVolumeSpecName: "istio-data") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:30.096930 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096865    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-credential-socket\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.096930 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096928    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-envoy\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.097235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.096971    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-socket\") pod \"29fb835d-2cb2-4c93-af1b-38a95406c14b\" (UID: \"29fb835d-2cb2-4c93-af1b-38a95406c14b\") "
Sep 16 13:57:30.097235 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097068    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:30.097356 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:30.097594 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097499    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.097594 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097522    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.097594 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097535    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.097594 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097547    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.097899 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.097724    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fb835d-2cb2-4c93-af1b-38a95406c14b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:57:30.099416 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.099384    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:57:30.099732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.099708    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-kube-api-access-jgj5r" (OuterVolumeSpecName: "kube-api-access-jgj5r") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "kube-api-access-jgj5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:30.099857 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.099833    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-token" (OuterVolumeSpecName: "istio-token") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:57:30.099950 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.099885    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29fb835d-2cb2-4c93-af1b-38a95406c14b" (UID: "29fb835d-2cb2-4c93-af1b-38a95406c14b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:57:30.198655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.198619    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29fb835d-2cb2-4c93-af1b-38a95406c14b-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.198655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.198649    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.198655 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.198658    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.198908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.198666    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgj5r\" (UniqueName: \"kubernetes.io/projected/29fb835d-2cb2-4c93-af1b-38a95406c14b-kube-api-access-jgj5r\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.198908 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.198677    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29fb835d-2cb2-4c93-af1b-38a95406c14b-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:57:30.850918 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.850887    2556 generic.go:356] "Generic (PLEG): container finished" podID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerID="d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca" exitCode=0
Sep 16 13:57:30.851140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.850937    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"
Sep 16 13:57:30.851140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.850969    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" event={"ID":"29fb835d-2cb2-4c93-af1b-38a95406c14b","Type":"ContainerDied","Data":"d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca"}
Sep 16 13:57:30.851140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.851019    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x" event={"ID":"29fb835d-2cb2-4c93-af1b-38a95406c14b","Type":"ContainerDied","Data":"9d9db73697284d94d81a86e0816a15aedfb00ad2ace523a3cb6cbf2e72db52cd"}
Sep 16 13:57:30.851140 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.851041    2556 scope.go:117] "RemoveContainer" containerID="d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca"
Sep 16 13:57:30.861688 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.861665    2556 scope.go:117] "RemoveContainer" containerID="bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"
Sep 16 13:57:30.870805 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.870777    2556 scope.go:117] "RemoveContainer" containerID="d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca"
Sep 16 13:57:30.871277 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:57:30.871247    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca\": container with ID starting with d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca not found: ID does not exist" containerID="d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca"
Sep 16 13:57:30.871403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.871311    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca"} err="failed to get container status \"d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca\": rpc error: code = NotFound desc = could not find container \"d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca\": container with ID starting with d036e47fb1711033861fca2ef4cbeb94e1126398b71cad2a1a52fa0a177b71ca not found: ID does not exist"
Sep 16 13:57:30.871403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.871336    2556 scope.go:117] "RemoveContainer" containerID="bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"
Sep 16 13:57:30.871746 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:57:30.871721    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460\": container with ID starting with bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460 not found: ID does not exist" containerID="bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"
Sep 16 13:57:30.871820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.871755    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460"} err="failed to get container status \"bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460\": rpc error: code = NotFound desc = could not find container \"bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460\": container with ID starting with bf07ca0a2adec6776fdf767a9dfdb4e6f69b2d7f0fa75fe746cc6dc9ac9d0460 not found: ID does not exist"
Sep 16 13:57:30.874577 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.874557    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"]
Sep 16 13:57:30.885838 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:30.885814    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7d96c7-openshift-default-6bfd8ccbb8-6rp5x"]
Sep 16 13:57:32.059270 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:32.059238    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" path="/var/lib/kubelet/pods/29fb835d-2cb2-4c93-af1b-38a95406c14b/volumes"
Sep 16 13:57:34.127678 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:34.127648    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:35.598345 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:35.598302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:36.392907 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:36.392870    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:37.053257 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:37.053224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:37.725795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:37.725761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:38.489032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:38.488997    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:38.800941 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:38.800856    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:39.531744 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.531705    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"]
Sep 16 13:57:39.532224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532180    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae901fb1-999e-4291-999d-bdfa6d479ded" containerName="main"
Sep 16 13:57:39.532224 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532217    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae901fb1-999e-4291-999d-bdfa6d479ded" containerName="main"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532231    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5a62d20-23ca-4388-86c1-e6d8247c83ea" containerName="main"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532237    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5a62d20-23ca-4388-86c1-e6d8247c83ea" containerName="main"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532250    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85386041-679a-4b3a-a873-6b1299a43674" containerName="main"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532256    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="85386041-679a-4b3a-a873-6b1299a43674" containerName="main"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532264    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532269    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532288    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" containerName="maas-api"
Sep 16 13:57:39.532348 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532294    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" containerName="maas-api"
Sep 16 13:57:39.532751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532367    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af3762dc-7d18-4eba-ad8d-3f06dc1dba62" containerName="istio-proxy"
Sep 16 13:57:39.532751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532376    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy"
Sep 16 13:57:39.532751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532383    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab5c28d4-bc34-47b7-9e3a-14620fd6f16a" containerName="maas-api"
Sep 16 13:57:39.532751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532388    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5a62d20-23ca-4388-86c1-e6d8247c83ea" containerName="main"
Sep 16 13:57:39.532751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532395    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="85386041-679a-4b3a-a873-6b1299a43674" containerName="main"
Sep 16 13:57:39.532751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.532401    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae901fb1-999e-4291-999d-bdfa6d479ded" containerName="main"
Sep 16 13:57:39.535582 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.535563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.537679 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.537659    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d6baf3-openshift-default-dockercfg-xb7cj\""
Sep 16 13:57:39.548033 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.548008    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"]
Sep 16 13:57:39.606149 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-istio-token\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/958c180e-cea0-440e-a11c-49801a197c16-istio-podinfo\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-data\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-socket\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-credential-socket\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606519 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-certs\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606615 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606594    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-envoy\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606652 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606636    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/958c180e-cea0-440e-a11c-49801a197c16-istiod-ca-cert\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.606732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.606716    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4jp\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-kube-api-access-lf4jp\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-istio-token\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707584    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/958c180e-cea0-440e-a11c-49801a197c16-istio-podinfo\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-data\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707647    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-socket\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707668    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-credential-socket\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707688    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-certs\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707732    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-envoy\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.707776 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707761    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/958c180e-cea0-440e-a11c-49801a197c16-istiod-ca-cert\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.708174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.707806    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4jp\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-kube-api-access-lf4jp\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.708174 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.708149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-socket\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.708288 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.708219    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-certs\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.708337 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.708323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-data\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.708397 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.708362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-credential-socket\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.708920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.708892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/958c180e-cea0-440e-a11c-49801a197c16-istiod-ca-cert\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.711762 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.711240    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-envoy\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.711762 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.711709    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/958c180e-cea0-440e-a11c-49801a197c16-istio-podinfo\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.726048 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.726012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-istio-token\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.726330 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.726308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4jp\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-kube-api-access-lf4jp\") pod \"e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") " pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:39.850084 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:39.849974    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:40.143788 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.143757    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"]
Sep 16 13:57:40.144953 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:57:40.144925    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958c180e_cea0_440e_a11c_49801a197c16.slice/crio-764b9b1deeafeba3f3d8c9349b0a1d00baa12044bab0957e024558badcc4131a WatchSource:0}: Error finding container 764b9b1deeafeba3f3d8c9349b0a1d00baa12044bab0957e024558badcc4131a: Status 404 returned error can't find the container with id 764b9b1deeafeba3f3d8c9349b0a1d00baa12044bab0957e024558badcc4131a
Sep 16 13:57:40.147323 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.147287    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:57:40.147403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.147375    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:57:40.147442 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.147408    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:57:40.326082 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.326037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:40.932032 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.931994    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" event={"ID":"958c180e-cea0-440e-a11c-49801a197c16","Type":"ContainerStarted","Data":"9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"}
Sep 16 13:57:40.932529 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.932053    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" event={"ID":"958c180e-cea0-440e-a11c-49801a197c16","Type":"ContainerStarted","Data":"764b9b1deeafeba3f3d8c9349b0a1d00baa12044bab0957e024558badcc4131a"}
Sep 16 13:57:40.956336 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:40.956281    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podStartSLOduration=1.9562666499999999 podStartE2EDuration="1.95626665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:40.954785444 +0000 UTC m=+1471.420722363" watchObservedRunningTime="2026-09-16 13:57:40.95626665 +0000 UTC m=+1471.422203637"
Sep 16 13:57:41.647607 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:41.647574    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:41.699874 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:41.699841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:41.850889 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:41.850851    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:41.856858 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:41.856831    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:41.938420 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:41.938391    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:41.939554 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:41.939531    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:57:42.186987 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:42.186949    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:42.863652 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:42.863615    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:42.972620 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:42.972585    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:43.066194 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:43.066158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:43.165812 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:43.165718    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:43.904407 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:43.904355    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:44.884572 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:44.884503    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:45.671465 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:45.671429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:46.341817 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:46.341775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:47.071952 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.071893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:47.174004 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.173972    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:47.746827 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.746790    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"]
Sep 16 13:57:47.747490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.747469    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy"
Sep 16 13:57:47.747578 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.747492    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy"
Sep 16 13:57:47.747636 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.747616    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="29fb835d-2cb2-4c93-af1b-38a95406c14b" containerName="istio-proxy"
Sep 16 13:57:47.751365 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.751339    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:47.753471 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.753441    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d6baf3-serving-cert\""
Sep 16 13:57:47.759429 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.759342    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"]
Sep 16 13:57:47.901412 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.901374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-metrics-certs\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:47.901606 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.901441    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ef235-facf-4f99-a82b-eb4a32080b57-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:47.901606 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.901512    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-maas-api-tls\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:47.901606 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:47.901567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42h9\" (UniqueName: \"kubernetes.io/projected/f67ef235-facf-4f99-a82b-eb4a32080b57-kube-api-access-s42h9\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.002843 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.002747    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-maas-api-tls\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.002843 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.002794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42h9\" (UniqueName: \"kubernetes.io/projected/f67ef235-facf-4f99-a82b-eb4a32080b57-kube-api-access-s42h9\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.003038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.002923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-metrics-certs\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.003038 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.002957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ef235-facf-4f99-a82b-eb4a32080b57-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.003624 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.003595    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ef235-facf-4f99-a82b-eb4a32080b57-openshift-service-ca\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.011122 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.005861    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-metrics-certs\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.011122 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.005906    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-maas-api-tls\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.012104 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.012063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42h9\" (UniqueName: \"kubernetes.io/projected/f67ef235-facf-4f99-a82b-eb4a32080b57-kube-api-access-s42h9\") pod \"maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.066139 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.066066    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:48.341434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.341394    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"]
Sep 16 13:57:48.343371 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:57:48.343343    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ef235_facf_4f99_a82b_eb4a32080b57.slice/crio-5e0f4dbd1e18841e72ad0d7994fbece7ae592a3b409d1b7fff1f6e243611f6ed WatchSource:0}: Error finding container 5e0f4dbd1e18841e72ad0d7994fbece7ae592a3b409d1b7fff1f6e243611f6ed: Status 404 returned error can't find the container with id 5e0f4dbd1e18841e72ad0d7994fbece7ae592a3b409d1b7fff1f6e243611f6ed
Sep 16 13:57:48.344650 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.344633    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:57:48.419794 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.419753    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"]
Sep 16 13:57:48.424716 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.424698    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.427163 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.427137    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d6baf3\""
Sep 16 13:57:48.427315 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.427240    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d6baf3-dockercfg-twxr5\""
Sep 16 13:57:48.434048 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.434023    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"]
Sep 16 13:57:48.509068 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.509028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0907a4d-632c-488d-8e6f-734da54def5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.509294 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.509119    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hspbl\" (UniqueName: \"kubernetes.io/projected/d0907a4d-632c-488d-8e6f-734da54def5b-kube-api-access-hspbl\") pod \"payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.609811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.609711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0907a4d-632c-488d-8e6f-734da54def5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.609811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.609793    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hspbl\" (UniqueName: \"kubernetes.io/projected/d0907a4d-632c-488d-8e6f-734da54def5b-kube-api-access-hspbl\") pod \"payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.610445 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.610424    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0907a4d-632c-488d-8e6f-734da54def5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.617464 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.617410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hspbl\" (UniqueName: \"kubernetes.io/projected/d0907a4d-632c-488d-8e6f-734da54def5b-kube-api-access-hspbl\") pod \"payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.738801 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.738775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:48.811489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.811413    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:48.911641 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.911541    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:57:48.986784 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.986744    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn" event={"ID":"f67ef235-facf-4f99-a82b-eb4a32080b57","Type":"ContainerStarted","Data":"944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7"}
Sep 16 13:57:48.987002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.986804    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn" event={"ID":"f67ef235-facf-4f99-a82b-eb4a32080b57","Type":"ContainerStarted","Data":"5e0f4dbd1e18841e72ad0d7994fbece7ae592a3b409d1b7fff1f6e243611f6ed"}
Sep 16 13:57:48.987002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:48.986868    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:49.004514 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:49.004458    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn" podStartSLOduration=1.663035592 podStartE2EDuration="2.00443902s" totalImagesPullingTime="341.403428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:49.001166348 +0000 UTC m=+1479.467103271" watchObservedRunningTime="2026-09-16 13:57:49.00443902 +0000 UTC m=+1479.470375962"
Sep 16 13:57:49.019804 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:49.019775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"]
Sep 16 13:57:49.020338 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:57:49.020317    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0907a4d_632c_488d_8e6f_734da54def5b.slice/crio-0a319d554e68dac63d584c85bcc51d115ef70f41a171ba8b8d10ff9a1731371f WatchSource:0}: Error finding container 0a319d554e68dac63d584c85bcc51d115ef70f41a171ba8b8d10ff9a1731371f: Status 404 returned error can't find the container with id 0a319d554e68dac63d584c85bcc51d115ef70f41a171ba8b8d10ff9a1731371f
Sep 16 13:57:49.996467 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:49.996427    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95" event={"ID":"d0907a4d-632c-488d-8e6f-734da54def5b","Type":"ContainerStarted","Data":"2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e"}
Sep 16 13:57:49.996948 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:49.996484    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95" event={"ID":"d0907a4d-632c-488d-8e6f-734da54def5b","Type":"ContainerStarted","Data":"0a319d554e68dac63d584c85bcc51d115ef70f41a171ba8b8d10ff9a1731371f"}
Sep 16 13:57:49.996948 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:49.996706    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:57:50.013751 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:50.013704    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95" podStartSLOduration=2.013689621 podStartE2EDuration="2.013689621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:57:50.010447238 +0000 UTC m=+1480.476384181" watchObservedRunningTime="2026-09-16 13:57:50.013689621 +0000 UTC m=+1480.479626528"
Sep 16 13:57:55.002616 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:55.002579    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:57:58.152107 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:58.152066    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:57:59.155256 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:57:59.155207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:00.344414 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:00.344365    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:01.005487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:01.005456    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:58:05.375887 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:05.375850    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:08.721300 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:08.721249    2556 patch_prober.go:28] interesting pod/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 13:58:08.721698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:08.721327    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 13:58:09.137569 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.137534    2556 generic.go:356] "Generic (PLEG): container finished" podID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerID="6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae" exitCode=0
Sep 16 13:58:09.137771 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.137618    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" event={"ID":"6920de26-b04b-4bc1-a5da-f4a20be5ee23","Type":"ContainerDied","Data":"6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"}
Sep 16 13:58:09.138276 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.138258    2556 scope.go:117] "RemoveContainer" containerID="6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"
Sep 16 13:58:09.139601 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.139570    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:58:09.139721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.139657    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:58:09.139721 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.139705    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:58:09.511167 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:09.511127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:10.145897 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.145863    2556 generic.go:356] "Generic (PLEG): container finished" podID="958c180e-cea0-440e-a11c-49801a197c16" containerID="9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996" exitCode=0
Sep 16 13:58:10.146372 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.145942    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" event={"ID":"958c180e-cea0-440e-a11c-49801a197c16","Type":"ContainerDied","Data":"9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"}
Sep 16 13:58:10.146483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.146459    2556 scope.go:117] "RemoveContainer" containerID="9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"
Sep 16 13:58:10.147850 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.147817    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:58:10.147940 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.147895    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:58:10.147986 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.147945    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 13:58:10.148243 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.148221    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" event={"ID":"6920de26-b04b-4bc1-a5da-f4a20be5ee23","Type":"ContainerStarted","Data":"042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f"}
Sep 16 13:58:10.493632 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:10.493583    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:11.048649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.048609    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:58:11.048832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.048651    2556 patch_prober.go:28] interesting pod/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 16 13:58:11.048832 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.048692    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 16 13:58:11.157187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.157145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" event={"ID":"958c180e-cea0-440e-a11c-49801a197c16","Type":"ContainerStarted","Data":"2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c"}
Sep 16 13:58:11.190890 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.190848    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:11.851398 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.851234    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:58:11.851758 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.851728    2556 patch_prober.go:28] interesting pod/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:58:11.851841 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.851782    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 16 13:58:11.938756 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:11.938714    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:58:12.048781 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.048745    2556 patch_prober.go:28] interesting pod/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 16 13:58:12.048965 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.048794    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 16 13:58:12.082711 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.082671    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:12.762455 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.762419    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:12.850989 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.850949    2556 patch_prober.go:28] interesting pod/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:58:12.851197 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.851005    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 16 13:58:12.975297 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:12.975258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:13.049213 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:13.049117    2556 patch_prober.go:28] interesting pod/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 16 13:58:13.049472 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:13.049444    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 16 13:58:13.574714 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:13.574688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:13.851218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:13.851083    2556 patch_prober.go:28] interesting pod/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:58:13.851218 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:13.851197    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 16 13:58:14.048827 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:14.048789    2556 patch_prober.go:28] interesting pod/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 16 13:58:14.049019 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:14.048857    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 16 13:58:14.282065 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:14.282037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:14.850894 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:14.850849    2556 patch_prober.go:28] interesting pod/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:58:14.851161 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:14.850911    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 16 13:58:15.049561 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:15.049508    2556 patch_prober.go:28] interesting pod/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 16 13:58:15.049958 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:15.049571    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 16 13:58:15.062715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:15.062683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:15.715614 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:15.715579    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:15.851359 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:15.851322    2556 patch_prober.go:28] interesting pod/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 16 13:58:15.851546 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:15.851391    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 16 13:58:16.052651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:16.052576    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:58:16.060056 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:16.060031    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:58:16.060226 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:16.060135    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:58:16.514529 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:16.514495    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:16.854931 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:16.854850    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:58:16.855891 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:16.855868    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:58:17.377823 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:17.377786    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:18.050291 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:18.050255    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:18.880707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:18.880674    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:19.637761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:19.637710    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:20.306761 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:20.306724    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:21.079854 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:21.079820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:21.245195 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:21.245148    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:58:21.726397 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:21.726354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:26.267127 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:26.267073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:31.841346 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:31.841265    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:31.940907 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:31.940874    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:32.040766 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:32.040732    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:32.144525 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:32.144443    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:32.390172 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:32.390127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:32.490625 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:32.490588    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:34.258403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:34.258372    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:34.611430 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:34.611355    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:35.292234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:35.292199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:36.096633 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:36.096597    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:36.829228 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:36.829201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:38.328425 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:38.328393    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:38.570409 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:38.570380    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"]
Sep 16 13:58:38.570612 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:38.570587    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy" containerID="cri-o://042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f" gracePeriod=30
Sep 16 13:58:38.973994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:38.973951    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:39.027138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:39.027058    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:39.282973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:39.282892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:40.079781 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:40.079745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:40.955596 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:40.955562    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:41.636472 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:41.636432    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:42.311023 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:42.310987    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:43.085614 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.085582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:43.768541 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.768469    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:43.832825 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.832726    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:58:43.919486 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919442    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-token\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919521    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-certs\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919561    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-data\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istiod-ca-cert\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919696 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919655    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-socket\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919718    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-podinfo\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919753    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-credential-socket\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhn7d\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-kube-api-access-hhn7d\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919802    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-envoy\") pod \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\" (UID: \"6920de26-b04b-4bc1-a5da-f4a20be5ee23\") "
Sep 16 13:58:43.919916 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919864    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:43.920212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919943    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-data" (OuterVolumeSpecName: "istio-data") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:43.920212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.919982    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:43.920212 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920047    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:43.920540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:58:43.920540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920417    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:43.920540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920436    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:43.920540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920448    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:43.920540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920461    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:43.920540 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.920474    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:43.922234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.922206    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:58:43.922234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.922212    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:58:43.922234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.922227    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-token" (OuterVolumeSpecName: "istio-token") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:43.922434 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:43.922209    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-kube-api-access-hhn7d" (OuterVolumeSpecName: "kube-api-access-hhn7d") pod "6920de26-b04b-4bc1-a5da-f4a20be5ee23" (UID: "6920de26-b04b-4bc1-a5da-f4a20be5ee23"). InnerVolumeSpecName "kube-api-access-hhn7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:44.021871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.021840    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:44.021871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.021869    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhn7d\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-kube-api-access-hhn7d\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:44.021871 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.021880    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:44.022137 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.021888    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6920de26-b04b-4bc1-a5da-f4a20be5ee23-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:44.399296 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.399265    2556 generic.go:356] "Generic (PLEG): container finished" podID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerID="042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f" exitCode=0
Sep 16 13:58:44.399689 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.399313    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"
Sep 16 13:58:44.399689 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.399346    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" event={"ID":"6920de26-b04b-4bc1-a5da-f4a20be5ee23","Type":"ContainerDied","Data":"042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f"}
Sep 16 13:58:44.399689 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.399394    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg" event={"ID":"6920de26-b04b-4bc1-a5da-f4a20be5ee23","Type":"ContainerDied","Data":"cf01dd814753979072b3debd86293c2fb56205c262560734fbbcb136c48d9972"}
Sep 16 13:58:44.399689 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.399417    2556 scope.go:117] "RemoveContainer" containerID="042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f"
Sep 16 13:58:44.409245 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.409229    2556 scope.go:117] "RemoveContainer" containerID="6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"
Sep 16 13:58:44.416498 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.416463    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"]
Sep 16 13:58:44.419267 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.419250    2556 scope.go:117] "RemoveContainer" containerID="042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f"
Sep 16 13:58:44.419613 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:58:44.419595    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f\": container with ID starting with 042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f not found: ID does not exist" containerID="042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f"
Sep 16 13:58:44.419703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.419631    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f"} err="failed to get container status \"042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f\": rpc error: code = NotFound desc = could not find container \"042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f\": container with ID starting with 042c7de87f6409381e0b32d194cd35b0696c85d619ea7659714d48d7ddffe65f not found: ID does not exist"
Sep 16 13:58:44.419703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.419654    2556 scope.go:117] "RemoveContainer" containerID="6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"
Sep 16 13:58:44.419703 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.419602    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-44f000-openshift-default-644d7f58fb-x2btg"]
Sep 16 13:58:44.419928 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:58:44.419911    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae\": container with ID starting with 6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae not found: ID does not exist" containerID="6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"
Sep 16 13:58:44.419982 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.419936    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae"} err="failed to get container status \"6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae\": rpc error: code = NotFound desc = could not find container \"6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae\": container with ID starting with 6d2f33bee4e322b579229e3766735253c0888ddeb2b93f525306ad3f238fceae not found: ID does not exist"
Sep 16 13:58:44.909108 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:44.909068    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:46.060256 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:46.060227    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" path="/var/lib/kubelet/pods/6920de26-b04b-4bc1-a5da-f4a20be5ee23/volumes"
Sep 16 13:58:54.420030 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.419995    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"]
Sep 16 13:58:54.420476 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.420293    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn" podUID="f67ef235-facf-4f99-a82b-eb4a32080b57" containerName="maas-api" containerID="cri-o://944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7" gracePeriod=30
Sep 16 13:58:54.444291 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.444266    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:54.452998 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.452973    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:58:54.479309 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.479278    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"]
Sep 16 13:58:54.479502 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.479478    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95" podUID="d0907a4d-632c-488d-8e6f-734da54def5b" containerName="payload-pre-processing" containerID="cri-o://2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e" gracePeriod=30
Sep 16 13:58:54.726041 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.726017    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:58:54.780222 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.780198    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:58:54.814637 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.814605    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:54.834261 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.834231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ef235-facf-4f99-a82b-eb4a32080b57-openshift-service-ca\") pod \"f67ef235-facf-4f99-a82b-eb4a32080b57\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") "
Sep 16 13:58:54.834421 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.834282    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s42h9\" (UniqueName: \"kubernetes.io/projected/f67ef235-facf-4f99-a82b-eb4a32080b57-kube-api-access-s42h9\") pod \"f67ef235-facf-4f99-a82b-eb4a32080b57\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") "
Sep 16 13:58:54.834421 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.834328    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-maas-api-tls\") pod \"f67ef235-facf-4f99-a82b-eb4a32080b57\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") "
Sep 16 13:58:54.834421 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.834407    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-metrics-certs\") pod \"f67ef235-facf-4f99-a82b-eb4a32080b57\" (UID: \"f67ef235-facf-4f99-a82b-eb4a32080b57\") "
Sep 16 13:58:54.834681 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.834648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ef235-facf-4f99-a82b-eb4a32080b57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f67ef235-facf-4f99-a82b-eb4a32080b57" (UID: "f67ef235-facf-4f99-a82b-eb4a32080b57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:58:54.834829 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.834800    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ef235-facf-4f99-a82b-eb4a32080b57-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:54.836651 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.836630    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ef235-facf-4f99-a82b-eb4a32080b57-kube-api-access-s42h9" (OuterVolumeSpecName: "kube-api-access-s42h9") pod "f67ef235-facf-4f99-a82b-eb4a32080b57" (UID: "f67ef235-facf-4f99-a82b-eb4a32080b57"). InnerVolumeSpecName "kube-api-access-s42h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:54.837109 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.837076    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f67ef235-facf-4f99-a82b-eb4a32080b57" (UID: "f67ef235-facf-4f99-a82b-eb4a32080b57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:58:54.837206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.837122    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f67ef235-facf-4f99-a82b-eb4a32080b57" (UID: "f67ef235-facf-4f99-a82b-eb4a32080b57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:58:54.879755 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.879706    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:54.936014 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.935980    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hspbl\" (UniqueName: \"kubernetes.io/projected/d0907a4d-632c-488d-8e6f-734da54def5b-kube-api-access-hspbl\") pod \"d0907a4d-632c-488d-8e6f-734da54def5b\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") "
Sep 16 13:58:54.936254 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.936127    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0907a4d-632c-488d-8e6f-734da54def5b-plugins-config-volume\") pod \"d0907a4d-632c-488d-8e6f-734da54def5b\" (UID: \"d0907a4d-632c-488d-8e6f-734da54def5b\") "
Sep 16 13:58:54.936467 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.936442    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0907a4d-632c-488d-8e6f-734da54def5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0907a4d-632c-488d-8e6f-734da54def5b" (UID: "d0907a4d-632c-488d-8e6f-734da54def5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:58:54.936550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.936531    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42h9\" (UniqueName: \"kubernetes.io/projected/f67ef235-facf-4f99-a82b-eb4a32080b57-kube-api-access-s42h9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:54.936609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.936557    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0907a4d-632c-488d-8e6f-734da54def5b-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:54.936609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.936572    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:54.936609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.936585    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ef235-facf-4f99-a82b-eb4a32080b57-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:54.938304 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:54.938256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0907a4d-632c-488d-8e6f-734da54def5b-kube-api-access-hspbl" (OuterVolumeSpecName: "kube-api-access-hspbl") pod "d0907a4d-632c-488d-8e6f-734da54def5b" (UID: "d0907a4d-632c-488d-8e6f-734da54def5b"). InnerVolumeSpecName "kube-api-access-hspbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:58:55.013163 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.013081    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:55.037378 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.037348    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hspbl\" (UniqueName: \"kubernetes.io/projected/d0907a4d-632c-488d-8e6f-734da54def5b-kube-api-access-hspbl\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:58:55.113988 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.113938    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:55.471128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.471075    2556 generic.go:356] "Generic (PLEG): container finished" podID="f67ef235-facf-4f99-a82b-eb4a32080b57" containerID="944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7" exitCode=0
Sep 16 13:58:55.471128 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.471132    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"
Sep 16 13:58:55.471618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.471132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn" event={"ID":"f67ef235-facf-4f99-a82b-eb4a32080b57","Type":"ContainerDied","Data":"944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7"}
Sep 16 13:58:55.471618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.471183    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn" event={"ID":"f67ef235-facf-4f99-a82b-eb4a32080b57","Type":"ContainerDied","Data":"5e0f4dbd1e18841e72ad0d7994fbece7ae592a3b409d1b7fff1f6e243611f6ed"}
Sep 16 13:58:55.471618 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.471207    2556 scope.go:117] "RemoveContainer" containerID="944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7"
Sep 16 13:58:55.472798 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.472771    2556 generic.go:356] "Generic (PLEG): container finished" podID="d0907a4d-632c-488d-8e6f-734da54def5b" containerID="2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e" exitCode=0
Sep 16 13:58:55.472925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.472809    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"
Sep 16 13:58:55.472925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.472841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95" event={"ID":"d0907a4d-632c-488d-8e6f-734da54def5b","Type":"ContainerDied","Data":"2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e"}
Sep 16 13:58:55.472925 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.472871    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95" event={"ID":"d0907a4d-632c-488d-8e6f-734da54def5b","Type":"ContainerDied","Data":"0a319d554e68dac63d584c85bcc51d115ef70f41a171ba8b8d10ff9a1731371f"}
Sep 16 13:58:55.485442 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.485418    2556 scope.go:117] "RemoveContainer" containerID="944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7"
Sep 16 13:58:55.485748 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:58:55.485731    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7\": container with ID starting with 944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7 not found: ID does not exist" containerID="944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7"
Sep 16 13:58:55.485830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.485753    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7"} err="failed to get container status \"944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7\": rpc error: code = NotFound desc = could not find container \"944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7\": container with ID starting with 944bc068f5d0854814866bf08d2ddbdaa96575e831ba213af0235e3d07fe79c7 not found: ID does not exist"
Sep 16 13:58:55.485830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.485769    2556 scope.go:117] "RemoveContainer" containerID="2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e"
Sep 16 13:58:55.495371 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.495355    2556 scope.go:117] "RemoveContainer" containerID="2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e"
Sep 16 13:58:55.495618 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:58:55.495601    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e\": container with ID starting with 2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e not found: ID does not exist" containerID="2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e"
Sep 16 13:58:55.495666 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.495624    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e"} err="failed to get container status \"2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e\": rpc error: code = NotFound desc = could not find container \"2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e\": container with ID starting with 2fbc102ebeb18c3075c055aad8c5b0833339f19657cfe9e310d95235a942555e not found: ID does not exist"
Sep 16 13:58:55.500558 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.500529    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"]
Sep 16 13:58:55.506550 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.506527    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d6baf3-5bdf79c85d-mzk95"]
Sep 16 13:58:55.516199 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.516168    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"]
Sep 16 13:58:55.519797 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.519767    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d6baf3-78c8f4f554-6jwdn"]
Sep 16 13:58:55.892059 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:55.891976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:56.059753 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:56.059720    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0907a4d-632c-488d-8e6f-734da54def5b" path="/var/lib/kubelet/pods/d0907a4d-632c-488d-8e6f-734da54def5b/volumes"
Sep 16 13:58:56.060157 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:56.060142    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67ef235-facf-4f99-a82b-eb4a32080b57" path="/var/lib/kubelet/pods/f67ef235-facf-4f99-a82b-eb4a32080b57/volumes"
Sep 16 13:58:56.580347 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:56.580314    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:57.368490 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:57.368453    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:58.051820 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:58.051779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:58.837459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:58.837421    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:58:59.738562 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:58:59.738528    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:00.086075 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:00.085990    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:00.765839 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:00.765806    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:01.349595 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.349565    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:01.450002 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.449962    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:01.520914 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.520879    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"]
Sep 16 13:59:01.521181 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.521155    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" containerID="cri-o://2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c" gracePeriod=30
Sep 16 13:59:01.545175 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.545144    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:01.616187 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.616075    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:01.858146 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.858105    2556 patch_prober.go:28] interesting pod/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 13:59:01.858562 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:01.858171    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 13:59:02.499649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:02.499615    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:03.237183 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:03.237146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:03.892278 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:03.892242    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:04.604105 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:04.604071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:04.727978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:04.727947    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"]
Sep 16 13:59:04.749542 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:04.749504    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"]
Sep 16 13:59:04.815884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:04.815846    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:05.367932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:05.367900    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:06.655565 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.655538    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:06.768649 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.768623    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:59:06.849796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.849700    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:06.855060 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855035    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf4jp\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-kube-api-access-lf4jp\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855067    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-istio-token\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855122    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-envoy\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855150    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/958c180e-cea0-440e-a11c-49801a197c16-istiod-ca-cert\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855196 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855193    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/958c180e-cea0-440e-a11c-49801a197c16-istio-podinfo\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855212    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-certs\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855234    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-socket\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855329    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-credential-socket\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855403 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855391    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-data\") pod \"958c180e-cea0-440e-a11c-49801a197c16\" (UID: \"958c180e-cea0-440e-a11c-49801a197c16\") "
Sep 16 13:59:06.855714 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855680    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/958c180e-cea0-440e-a11c-49801a197c16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:59:06.855801 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855680    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:06.855939 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.855916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-data" (OuterVolumeSpecName: "istio-data") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:06.856119 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.856079    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:06.856176 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.856160    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:06.857567 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.857543    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-kube-api-access-lf4jp" (OuterVolumeSpecName: "kube-api-access-lf4jp") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "kube-api-access-lf4jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:06.857567 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.857547    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/958c180e-cea0-440e-a11c-49801a197c16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:59:06.857961 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.857945    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-istio-token" (OuterVolumeSpecName: "istio-token") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:06.857961 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.857950    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "958c180e-cea0-440e-a11c-49801a197c16" (UID: "958c180e-cea0-440e-a11c-49801a197c16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:06.904144 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.904105    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:06.956483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956448    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956475    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/958c180e-cea0-440e-a11c-49801a197c16-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956484    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/958c180e-cea0-440e-a11c-49801a197c16-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956494    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956505    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956513    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956522    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/958c180e-cea0-440e-a11c-49801a197c16-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956530    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4jp\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-kube-api-access-lf4jp\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:06.956720 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:06.956538    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/958c180e-cea0-440e-a11c-49801a197c16-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:07.551154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.551120    2556 generic.go:356] "Generic (PLEG): container finished" podID="958c180e-cea0-440e-a11c-49801a197c16" containerID="2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c" exitCode=0
Sep 16 13:59:07.551154 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.551157    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"
Sep 16 13:59:07.551376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.551189    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" event={"ID":"958c180e-cea0-440e-a11c-49801a197c16","Type":"ContainerDied","Data":"2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c"}
Sep 16 13:59:07.551376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.551240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768" event={"ID":"958c180e-cea0-440e-a11c-49801a197c16","Type":"ContainerDied","Data":"764b9b1deeafeba3f3d8c9349b0a1d00baa12044bab0957e024558badcc4131a"}
Sep 16 13:59:07.551376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.551265    2556 scope.go:117] "RemoveContainer" containerID="2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c"
Sep 16 13:59:07.562657 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.562632    2556 scope.go:117] "RemoveContainer" containerID="9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"
Sep 16 13:59:07.576338 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.576289    2556 scope.go:117] "RemoveContainer" containerID="2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c"
Sep 16 13:59:07.576643 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:07.576621    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c\": container with ID starting with 2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c not found: ID does not exist" containerID="2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c"
Sep 16 13:59:07.576711 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.576652    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c"} err="failed to get container status \"2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c\": rpc error: code = NotFound desc = could not find container \"2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c\": container with ID starting with 2a7f58ccb9731e537a63df16111c9c8e9d4cca83ce2faa0a918f7a8ae74a715c not found: ID does not exist"
Sep 16 13:59:07.576711 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.576671    2556 scope.go:117] "RemoveContainer" containerID="9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"
Sep 16 13:59:07.576811 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.576789    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"]
Sep 16 13:59:07.576957 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:07.576939    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996\": container with ID starting with 9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996 not found: ID does not exist" containerID="9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"
Sep 16 13:59:07.577017 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.576967    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996"} err="failed to get container status \"9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996\": rpc error: code = NotFound desc = could not find container \"9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996\": container with ID starting with 9fb26952468a18c0b9c82577b74fadd2f4affacf758fafe90498f67da67d2996 not found: ID does not exist"
Sep 16 13:59:07.579379 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:07.579358    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d6baf3-openshift-default-69fbd4974-z6768"]
Sep 16 13:59:08.060063 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:08.060022    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="958c180e-cea0-440e-a11c-49801a197c16" path="/var/lib/kubelet/pods/958c180e-cea0-440e-a11c-49801a197c16/volumes"
Sep 16 13:59:10.020973 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:10.020941    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:12.022331 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:12.022294    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:12.726863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:12.726835    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:12.827384 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:12.827354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:12.918441 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:12.918404    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:12.981055 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:12.980975    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:13.284021 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:13.283928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:13.381305 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:13.381269    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:14.088792 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:14.088741    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:14.874837 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:14.874809    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.420518 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.420484    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"]
Sep 16 13:59:15.420768 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.420722    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks" podUID="52c85d93-7a37-4e2b-8143-0a99559cb8e0" containerName="maas-api" containerID="cri-o://c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113" gracePeriod=30
Sep 16 13:59:15.450956 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.450915    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.515437 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.515402    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.568485 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.568450    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.669660 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.669625    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.727754 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.727701    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.872715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.872685    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:15.881230 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.881210    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:59:15.948385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.948293    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4grtk\" (UniqueName: \"kubernetes.io/projected/52c85d93-7a37-4e2b-8143-0a99559cb8e0-kube-api-access-4grtk\") pod \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") "
Sep 16 13:59:15.948385 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.948347    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-maas-api-tls\") pod \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") "
Sep 16 13:59:15.948609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.948397    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-metrics-certs\") pod \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") "
Sep 16 13:59:15.948609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.948476    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c85d93-7a37-4e2b-8143-0a99559cb8e0-openshift-service-ca\") pod \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\" (UID: \"52c85d93-7a37-4e2b-8143-0a99559cb8e0\") "
Sep 16 13:59:15.948978 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.948940    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c85d93-7a37-4e2b-8143-0a99559cb8e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52c85d93-7a37-4e2b-8143-0a99559cb8e0" (UID: "52c85d93-7a37-4e2b-8143-0a99559cb8e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:59:15.950739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.950712    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c85d93-7a37-4e2b-8143-0a99559cb8e0-kube-api-access-4grtk" (OuterVolumeSpecName: "kube-api-access-4grtk") pod "52c85d93-7a37-4e2b-8143-0a99559cb8e0" (UID: "52c85d93-7a37-4e2b-8143-0a99559cb8e0"). InnerVolumeSpecName "kube-api-access-4grtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:15.950847 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.950745    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52c85d93-7a37-4e2b-8143-0a99559cb8e0" (UID: "52c85d93-7a37-4e2b-8143-0a99559cb8e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:59:15.950847 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:15.950794    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52c85d93-7a37-4e2b-8143-0a99559cb8e0" (UID: "52c85d93-7a37-4e2b-8143-0a99559cb8e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:59:16.049869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.049819    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52c85d93-7a37-4e2b-8143-0a99559cb8e0-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:16.049869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.049856    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4grtk\" (UniqueName: \"kubernetes.io/projected/52c85d93-7a37-4e2b-8143-0a99559cb8e0-kube-api-access-4grtk\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:16.049869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.049865    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:16.049869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.049873    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52c85d93-7a37-4e2b-8143-0a99559cb8e0-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:16.295249 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.295161    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:16.610487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.610407    2556 generic.go:356] "Generic (PLEG): container finished" podID="52c85d93-7a37-4e2b-8143-0a99559cb8e0" containerID="c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113" exitCode=0
Sep 16 13:59:16.610487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.610450    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"
Sep 16 13:59:16.610487 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.610455    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks" event={"ID":"52c85d93-7a37-4e2b-8143-0a99559cb8e0","Type":"ContainerDied","Data":"c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113"}
Sep 16 13:59:16.610715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.610496    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks" event={"ID":"52c85d93-7a37-4e2b-8143-0a99559cb8e0","Type":"ContainerDied","Data":"1d160266186e1e160d4c2caec90df18c10663c0d5739fbed59a8e9f47a355b63"}
Sep 16 13:59:16.610715 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.610517    2556 scope.go:117] "RemoveContainer" containerID="c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113"
Sep 16 13:59:16.619827 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.619805    2556 scope.go:117] "RemoveContainer" containerID="c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113"
Sep 16 13:59:16.620119 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:16.620081    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113\": container with ID starting with c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113 not found: ID does not exist" containerID="c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113"
Sep 16 13:59:16.620205 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.620126    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113"} err="failed to get container status \"c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113\": rpc error: code = NotFound desc = could not find container \"c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113\": container with ID starting with c610f52ac8ea36ac46b62a555110ccaadc2e04417959a4f134de56694bedb113 not found: ID does not exist"
Sep 16 13:59:16.626281 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.626258    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"]
Sep 16 13:59:16.629501 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.629481    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-fc8c65-584bb6c7f8-q7nks"]
Sep 16 13:59:16.895157 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:16.895067    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:17.531408 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:17.531369    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:18.059583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:18.059551    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c85d93-7a37-4e2b-8143-0a99559cb8e0" path="/var/lib/kubelet/pods/52c85d93-7a37-4e2b-8143-0a99559cb8e0/volumes"
Sep 16 13:59:18.196138 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:18.196108    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:18.991160 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:18.991128    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:19.576446 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:19.576412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:19.675898 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:19.675872    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:19.748787 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:19.748750    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz" podUID="e41cfe8c-342a-4309-b452-fd2e8ca95946" containerName="main" containerID="cri-o://bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4" gracePeriod=45
Sep 16 13:59:19.770709 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:19.770674    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh" podUID="bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" containerName="main" containerID="cri-o://d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea" gracePeriod=45
Sep 16 13:59:19.820559 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:19.820522    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:20.065486 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.065447    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:59:20.068830 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.068807    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:59:20.194289 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194252    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tls-certs\") pod \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") "
Sep 16 13:59:20.194469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194301    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-dshm\") pod \"e41cfe8c-342a-4309-b452-fd2e8ca95946\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") "
Sep 16 13:59:20.194469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194390    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41cfe8c-342a-4309-b452-fd2e8ca95946-tls-certs\") pod \"e41cfe8c-342a-4309-b452-fd2e8ca95946\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") "
Sep 16 13:59:20.194469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194426    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x58rz\" (UniqueName: \"kubernetes.io/projected/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-kube-api-access-x58rz\") pod \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") "
Sep 16 13:59:20.194469 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194468    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-home\") pod \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") "
Sep 16 13:59:20.194707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194493    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tmp-dir\") pod \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") "
Sep 16 13:59:20.194707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194544    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lnbd\" (UniqueName: \"kubernetes.io/projected/e41cfe8c-342a-4309-b452-fd2e8ca95946-kube-api-access-4lnbd\") pod \"e41cfe8c-342a-4309-b452-fd2e8ca95946\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") "
Sep 16 13:59:20.194707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194594    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-model-cache\") pod \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") "
Sep 16 13:59:20.194707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194619    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-home\") pod \"e41cfe8c-342a-4309-b452-fd2e8ca95946\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") "
Sep 16 13:59:20.194707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194671    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-model-cache\") pod \"e41cfe8c-342a-4309-b452-fd2e8ca95946\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") "
Sep 16 13:59:20.194707 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194705    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-dshm\") pod \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\" (UID: \"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987\") "
Sep 16 13:59:20.194999 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.194755    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-tmp-dir\") pod \"e41cfe8c-342a-4309-b452-fd2e8ca95946\" (UID: \"e41cfe8c-342a-4309-b452-fd2e8ca95946\") "
Sep 16 13:59:20.195058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195008    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" (UID: "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.195255 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195235    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.195372 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195243    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-model-cache" (OuterVolumeSpecName: "model-cache") pod "e41cfe8c-342a-4309-b452-fd2e8ca95946" (UID: "e41cfe8c-342a-4309-b452-fd2e8ca95946"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.195627 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195259    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e41cfe8c-342a-4309-b452-fd2e8ca95946" (UID: "e41cfe8c-342a-4309-b452-fd2e8ca95946"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.195712 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195190    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" (UID: "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.195796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195559    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-home" (OuterVolumeSpecName: "home") pod "e41cfe8c-342a-4309-b452-fd2e8ca95946" (UID: "e41cfe8c-342a-4309-b452-fd2e8ca95946"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.195893 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.195829    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-home" (OuterVolumeSpecName: "home") pod "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" (UID: "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.197293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.197255    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" (UID: "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:59:20.197293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.197288    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-dshm" (OuterVolumeSpecName: "dshm") pod "e41cfe8c-342a-4309-b452-fd2e8ca95946" (UID: "e41cfe8c-342a-4309-b452-fd2e8ca95946"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.197828 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.197796    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41cfe8c-342a-4309-b452-fd2e8ca95946-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e41cfe8c-342a-4309-b452-fd2e8ca95946" (UID: "e41cfe8c-342a-4309-b452-fd2e8ca95946"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:59:20.197947 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.197887    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-dshm" (OuterVolumeSpecName: "dshm") pod "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" (UID: "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:20.198440 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.198419    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-kube-api-access-x58rz" (OuterVolumeSpecName: "kube-api-access-x58rz") pod "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" (UID: "bf1b7c57-3f82-40da-8cc7-b8f05dd4d987"). InnerVolumeSpecName "kube-api-access-x58rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:20.198508 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.198462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41cfe8c-342a-4309-b452-fd2e8ca95946-kube-api-access-4lnbd" (OuterVolumeSpecName: "kube-api-access-4lnbd") pod "e41cfe8c-342a-4309-b452-fd2e8ca95946" (UID: "e41cfe8c-342a-4309-b452-fd2e8ca95946"). InnerVolumeSpecName "kube-api-access-4lnbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:20.296842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296809    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.296842 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296842    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296857    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296869    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296878    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296885    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41cfe8c-342a-4309-b452-fd2e8ca95946-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296894    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41cfe8c-342a-4309-b452-fd2e8ca95946-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296902    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58rz\" (UniqueName: \"kubernetes.io/projected/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-kube-api-access-x58rz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296910    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296918    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.297058 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.296925    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lnbd\" (UniqueName: \"kubernetes.io/projected/e41cfe8c-342a-4309-b452-fd2e8ca95946-kube-api-access-4lnbd\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:20.351124 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.351078    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:20.637053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.636957    2556 generic.go:356] "Generic (PLEG): container finished" podID="e41cfe8c-342a-4309-b452-fd2e8ca95946" containerID="bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4" exitCode=0
Sep 16 13:59:20.637053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.637003    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"
Sep 16 13:59:20.637053 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.637038    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz" event={"ID":"e41cfe8c-342a-4309-b452-fd2e8ca95946","Type":"ContainerDied","Data":"bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4"}
Sep 16 13:59:20.637619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.637082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz" event={"ID":"e41cfe8c-342a-4309-b452-fd2e8ca95946","Type":"ContainerDied","Data":"155ab3535e4224f8ef7017327ac4fa78fadf769fb9b532ed1f3fd616b43d57ff"}
Sep 16 13:59:20.637619 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.637121    2556 scope.go:117] "RemoveContainer" containerID="bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4"
Sep 16 13:59:20.638609 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.638583    2556 generic.go:356] "Generic (PLEG): container finished" podID="bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" containerID="d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea" exitCode=0
Sep 16 13:59:20.638691 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.638615    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh" event={"ID":"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987","Type":"ContainerDied","Data":"d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea"}
Sep 16 13:59:20.638691 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.638628    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"
Sep 16 13:59:20.638691 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.638647    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh" event={"ID":"bf1b7c57-3f82-40da-8cc7-b8f05dd4d987","Type":"ContainerDied","Data":"bdd5f911c303553ddc2c04760a24b11e76f0647675cad6a12b391a2cf975cea8"}
Sep 16 13:59:20.647994 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.647972    2556 scope.go:117] "RemoveContainer" containerID="bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4"
Sep 16 13:59:20.648376 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:20.648322    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4\": container with ID starting with bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4 not found: ID does not exist" containerID="bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4"
Sep 16 13:59:20.648376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.648346    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4"} err="failed to get container status \"bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4\": rpc error: code = NotFound desc = could not find container \"bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4\": container with ID starting with bc0604c02cd31e72c81ceb131dcd66b1c6b71c8bb1795fe626ad4c77f5b435a4 not found: ID does not exist"
Sep 16 13:59:20.648376 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.648369    2556 scope.go:117] "RemoveContainer" containerID="d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea"
Sep 16 13:59:20.657011 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.656989    2556 scope.go:117] "RemoveContainer" containerID="d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea"
Sep 16 13:59:20.657480 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:20.657449    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea\": container with ID starting with d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea not found: ID does not exist" containerID="d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea"
Sep 16 13:59:20.657612 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.657488    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea"} err="failed to get container status \"d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea\": rpc error: code = NotFound desc = could not find container \"d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea\": container with ID starting with d9b6eecf641c0e659a55a65282b0df274e9440018256cd32e37bbf60881401ea not found: ID does not exist"
Sep 16 13:59:20.660425 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.660403    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"]
Sep 16 13:59:20.664875 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.664856    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/distinct-w4-fc8c65-kserve-7f8dc785c6-25lkz"]
Sep 16 13:59:20.674349 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.674323    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"]
Sep 16 13:59:20.676822 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:20.676800    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc8c65/simulator-w4-fc8c65-kserve-b97895d96-9nwvh"]
Sep 16 13:59:21.275778 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:21.275740    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:21.946920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:21.946879    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:22.040632 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:22.040592    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:22.064439 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:22.061748    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" path="/var/lib/kubelet/pods/bf1b7c57-3f82-40da-8cc7-b8f05dd4d987/volumes"
Sep 16 13:59:22.064439 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:22.062367    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41cfe8c-342a-4309-b452-fd2e8ca95946" path="/var/lib/kubelet/pods/e41cfe8c-342a-4309-b452-fd2e8ca95946/volumes"
Sep 16 13:59:22.137884 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:22.137848    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:59:22.204969 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:22.204862    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:22.745250 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:22.745216    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:23.632698 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:23.632661    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:25.447507 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:25.447455    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:25.548031 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:25.548001    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:26.587303 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:26.587267    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:30.649474 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:30.649423    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:30.711926 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:30.711887    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:30.912708 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:30.912633    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:30.974424 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:30.974392    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:31.258917 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:31.258891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:31.360328 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:31.360290    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:31.978362 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:31.978318    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:32.540583 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:32.540548    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:32.663340 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:32.663299    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:36.765781 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:36.765745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:36.913470 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:36.913405    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:37.017549 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.017461    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"]
Sep 16 13:59:37.017719 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.017688    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy" containerID="cri-o://c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b" gracePeriod=30
Sep 16 13:59:37.042677 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.042646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:37.144208 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144174    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-a77e69"]
Sep 16 13:59:37.144593 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144581    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144595    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144609    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" containerName="main"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144615    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" containerName="main"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144623    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c85d93-7a37-4e2b-8143-0a99559cb8e0" containerName="maas-api"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144629    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c85d93-7a37-4e2b-8143-0a99559cb8e0" containerName="maas-api"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144635    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy"
Sep 16 13:59:37.144639 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144640    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144649    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144654    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144664    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41cfe8c-342a-4309-b452-fd2e8ca95946" containerName="main"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144669    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41cfe8c-342a-4309-b452-fd2e8ca95946" containerName="main"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144677    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67ef235-facf-4f99-a82b-eb4a32080b57" containerName="maas-api"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144681    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67ef235-facf-4f99-a82b-eb4a32080b57" containerName="maas-api"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144694    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0907a4d-632c-488d-8e6f-734da54def5b" containerName="payload-pre-processing"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144701    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0907a4d-632c-488d-8e6f-734da54def5b" containerName="payload-pre-processing"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144761    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41cfe8c-342a-4309-b452-fd2e8ca95946" containerName="main"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144769    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144776    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144782    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0907a4d-632c-488d-8e6f-734da54def5b" containerName="payload-pre-processing"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144787    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c85d93-7a37-4e2b-8143-0a99559cb8e0" containerName="maas-api"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144793    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf1b7c57-3f82-40da-8cc7-b8f05dd4d987" containerName="main"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144799    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="958c180e-cea0-440e-a11c-49801a197c16" containerName="istio-proxy"
Sep 16 13:59:37.144853 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.144805    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67ef235-facf-4f99-a82b-eb4a32080b57" containerName="maas-api"
Sep 16 13:59:37.154739 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.154704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:37.156778 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.156750    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cmn8d\""
Sep 16 13:59:37.157796 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.157775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-a77e69"]
Sep 16 13:59:37.264157 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.264121    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xchnh\" (UniqueName: \"kubernetes.io/projected/e7a63768-65b7-4f37-9bcf-29a902ec23fa-kube-api-access-xchnh\") pod \"test-curl-7695-a77e69\" (UID: \"e7a63768-65b7-4f37-9bcf-29a902ec23fa\") " pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:37.365521 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.365411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xchnh\" (UniqueName: \"kubernetes.io/projected/e7a63768-65b7-4f37-9bcf-29a902ec23fa-kube-api-access-xchnh\") pod \"test-curl-7695-a77e69\" (UID: \"e7a63768-65b7-4f37-9bcf-29a902ec23fa\") " pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:37.373701 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.373668    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xchnh\" (UniqueName: \"kubernetes.io/projected/e7a63768-65b7-4f37-9bcf-29a902ec23fa-kube-api-access-xchnh\") pod \"test-curl-7695-a77e69\" (UID: \"e7a63768-65b7-4f37-9bcf-29a902ec23fa\") " pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:37.465238 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.465202    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:37.735483 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.735452    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-a77e69"]
Sep 16 13:59:37.736975 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:59:37.736949    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a63768_65b7_4f37_9bcf_29a902ec23fa.slice/crio-cd16a964172751875d4704b434ac5367c75089ed2b3b5f0e509fa158a0bdab0d WatchSource:0}: Error finding container cd16a964172751875d4704b434ac5367c75089ed2b3b5f0e509fa158a0bdab0d: Status 404 returned error can't find the container with id cd16a964172751875d4704b434ac5367c75089ed2b3b5f0e509fa158a0bdab0d
Sep 16 13:59:37.751293 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:37.751260    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-a77e69" event={"ID":"e7a63768-65b7-4f37-9bcf-29a902ec23fa","Type":"ContainerStarted","Data":"cd16a964172751875d4704b434ac5367c75089ed2b3b5f0e509fa158a0bdab0d"}
Sep 16 13:59:38.759076 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:38.759041    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-a77e69" event={"ID":"e7a63768-65b7-4f37-9bcf-29a902ec23fa","Type":"ContainerStarted","Data":"8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4"}
Sep 16 13:59:38.773692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:38.773647    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-a77e69" podStartSLOduration=1.773634398 podStartE2EDuration="1.773634398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:38.771459693 +0000 UTC m=+1589.237396602" watchObservedRunningTime="2026-09-16 13:59:38.773634398 +0000 UTC m=+1589.239571334"
Sep 16 13:59:39.367106 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:39.367052    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-a77e69"]
Sep 16 13:59:39.371386 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:39.371363    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-a77e69"]
Sep 16 13:59:39.765746 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:39.765714    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-a77e69" container="test-curl-7695-a77e69"
Sep 16 13:59:40.717318 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:40.717291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:40.771692 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:40.771650    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-a77e69" podUID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" containerName="test-curl-7695-a77e69" containerID="cri-o://8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4" gracePeriod=2
Sep 16 13:59:40.843737 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:40.843698    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:59:41.672375 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:41.672347    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:42.261303 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.261283    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:59:42.417206 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417183    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-podinfo\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-token\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417364 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417290    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-credential-socket\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417365    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsmjw\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-kube-api-access-hsmjw\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417412    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-certs\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417453    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-socket\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417489 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417483    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-data\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417704 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417513    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-envoy\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417704 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417537    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a453e-4175-4b96-8d9e-9ef0da349fe9-istiod-ca-cert\") pod \"052a453e-4175-4b96-8d9e-9ef0da349fe9\" (UID: \"052a453e-4175-4b96-8d9e-9ef0da349fe9\") "
Sep 16 13:59:42.417704 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417662    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:42.417704 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417680    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:42.417920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417733    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-data" (OuterVolumeSpecName: "istio-data") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:42.417920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417798    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:42.417920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417897    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.417920 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417915    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.418166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417929    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.418166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417942    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.418166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.417967    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052a453e-4175-4b96-8d9e-9ef0da349fe9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:59:42.419690 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.419657    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 13:59:42.420179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.419927    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-token" (OuterVolumeSpecName: "istio-token") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:42.420179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.419983    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:59:42.420179 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.419981    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-kube-api-access-hsmjw" (OuterVolumeSpecName: "kube-api-access-hsmjw") pod "052a453e-4175-4b96-8d9e-9ef0da349fe9" (UID: "052a453e-4175-4b96-8d9e-9ef0da349fe9"). InnerVolumeSpecName "kube-api-access-hsmjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:42.518773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.518741    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmjw\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-kube-api-access-hsmjw\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.518773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.518768    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.518773 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.518777    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/052a453e-4175-4b96-8d9e-9ef0da349fe9-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.518992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.518785    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.518992 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.518793    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/052a453e-4175-4b96-8d9e-9ef0da349fe9-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:42.785079 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.785053    2556 generic.go:356] "Generic (PLEG): container finished" podID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerID="c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b" exitCode=0
Sep 16 13:59:42.785210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.785116    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"
Sep 16 13:59:42.785210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.785129    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" event={"ID":"052a453e-4175-4b96-8d9e-9ef0da349fe9","Type":"ContainerDied","Data":"c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b"}
Sep 16 13:59:42.785210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.785184    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x" event={"ID":"052a453e-4175-4b96-8d9e-9ef0da349fe9","Type":"ContainerDied","Data":"212e1f4f7c49262d2689b8f2fddd391ba93e30c3758be49bc3e6fa6dc24fa33d"}
Sep 16 13:59:42.785210 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.785208    2556 scope.go:117] "RemoveContainer" containerID="c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b"
Sep 16 13:59:42.827182 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.827165    2556 scope.go:117] "RemoveContainer" containerID="164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"
Sep 16 13:59:42.837492 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.837467    2556 scope.go:117] "RemoveContainer" containerID="c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b"
Sep 16 13:59:42.837676 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.837653    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"]
Sep 16 13:59:42.837883 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:42.837865    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b\": container with ID starting with c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b not found: ID does not exist" containerID="c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b"
Sep 16 13:59:42.837932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.837893    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b"} err="failed to get container status \"c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b\": rpc error: code = NotFound desc = could not find container \"c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b\": container with ID starting with c4f5fc4fea2a11e49fdf24b348f052539fbb671d60dd9a4c520ee7835ce2054b not found: ID does not exist"
Sep 16 13:59:42.837932 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.837911    2556 scope.go:117] "RemoveContainer" containerID="164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"
Sep 16 13:59:42.838251 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:42.838223    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c\": container with ID starting with 164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c not found: ID does not exist" containerID="164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"
Sep 16 13:59:42.838251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.838255    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c"} err="failed to get container status \"164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c\": rpc error: code = NotFound desc = could not find container \"164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c\": container with ID starting with 164a252d5c7a63fc0d627fb7911946f3dee431b63fb018168ff8cb9b4d21132c not found: ID does not exist"
Sep 16 13:59:42.842166 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.842139    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-284b60-openshift-default-7dc65485d4-k684x"]
Sep 16 13:59:42.937066 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:42.937034    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:43.022902 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.022867    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xchnh\" (UniqueName: \"kubernetes.io/projected/e7a63768-65b7-4f37-9bcf-29a902ec23fa-kube-api-access-xchnh\") pod \"e7a63768-65b7-4f37-9bcf-29a902ec23fa\" (UID: \"e7a63768-65b7-4f37-9bcf-29a902ec23fa\") "
Sep 16 13:59:43.025204 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.025140    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a63768-65b7-4f37-9bcf-29a902ec23fa-kube-api-access-xchnh" (OuterVolumeSpecName: "kube-api-access-xchnh") pod "e7a63768-65b7-4f37-9bcf-29a902ec23fa" (UID: "e7a63768-65b7-4f37-9bcf-29a902ec23fa"). InnerVolumeSpecName "kube-api-access-xchnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:59:43.124205 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.124169    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xchnh\" (UniqueName: \"kubernetes.io/projected/e7a63768-65b7-4f37-9bcf-29a902ec23fa-kube-api-access-xchnh\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 13:59:43.796886 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.796855    2556 generic.go:356] "Generic (PLEG): container finished" podID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" containerID="8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4" exitCode=137
Sep 16 13:59:43.797327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.796898    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:43.797327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.796906    2556 scope.go:117] "RemoveContainer" containerID="8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4"
Sep 16 13:59:43.806968 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.806946    2556 scope.go:117] "RemoveContainer" containerID="8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4"
Sep 16 13:59:43.807228 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:43.807201    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7695-a77e69\" is forbidden: User \"system:node:ip-10-0-133-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-251.ec2.internal' and this object" podUID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" pod="opendatahub/test-curl-7695-a77e69"
Sep 16 13:59:43.807300 ip-10-0-133-251 kubenswrapper[2556]: E0916 13:59:43.807280    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4\": container with ID starting with 8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4 not found: ID does not exist" containerID="8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4"
Sep 16 13:59:43.807347 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:43.807308    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4"} err="failed to get container status \"8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4\": rpc error: code = NotFound desc = could not find container \"8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4\": container with ID starting with 8620547926ad215c464e7f4fbc31bb696405c4ed4e0ba7595858e775818601f4 not found: ID does not exist"
Sep 16 13:59:44.059459 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:44.059382    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" path="/var/lib/kubelet/pods/052a453e-4175-4b96-8d9e-9ef0da349fe9/volumes"
Sep 16 13:59:44.059733 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:44.059720    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" path="/var/lib/kubelet/pods/e7a63768-65b7-4f37-9bcf-29a902ec23fa/volumes"
Sep 16 13:59:49.331505 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:49.331472    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 13:59:51.147935 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.147885    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"]
Sep 16 13:59:51.148564 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148543    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148566    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148595    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148605    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148624    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" containerName="test-curl-7695-a77e69"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148632    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" containerName="test-curl-7695-a77e69"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148647    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy"
Sep 16 13:59:51.148656 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148656    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy"
Sep 16 13:59:51.148869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148758    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a63768-65b7-4f37-9bcf-29a902ec23fa" containerName="test-curl-7695-a77e69"
Sep 16 13:59:51.148869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148773    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy"
Sep 16 13:59:51.148869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148785    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="052a453e-4175-4b96-8d9e-9ef0da349fe9" containerName="istio-proxy"
Sep 16 13:59:51.148869 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.148796    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6920de26-b04b-4bc1-a5da-f4a20be5ee23" containerName="istio-proxy"
Sep 16 13:59:51.152211 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.152189    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.154863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.154576    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2284bd\"/\"openshift-service-ca.crt\""
Sep 16 13:59:51.154863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.154760    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2284bd\"/\"default-dockercfg-pds95\""
Sep 16 13:59:51.154863 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.154830    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2284bd\"/\"kube-root-ca.crt\""
Sep 16 13:59:51.155115 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.154910    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2284bd\"/\"sub-test-model-2284bd-kserve-self-signed-certs\""
Sep 16 13:59:51.161234 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.161214    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"]
Sep 16 13:59:51.183927 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.183899    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 13:59:51.193077 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.193045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tls-certs\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.193251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.193148    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-dshm\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.193251 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.193169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvzq\" (UniqueName: \"kubernetes.io/projected/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-kube-api-access-zhvzq\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.193327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.193245    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tmp-dir\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.193327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.193294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-home\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.193327 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.193314    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-model-cache\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294204 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294166    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tmp-dir\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294400 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294225    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-home\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294400 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294252    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-model-cache\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294400 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294322    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tls-certs\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-dshm\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294575 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvzq\" (UniqueName: \"kubernetes.io/projected/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-kube-api-access-zhvzq\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294666 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294609    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tmp-dir\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294719 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-home\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.294719 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.294676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-model-cache\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.296795 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.296767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-dshm\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.296923 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.296907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tls-certs\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.301947 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.301924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvzq\" (UniqueName: \"kubernetes.io/projected/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-kube-api-access-zhvzq\") pod \"sub-test-model-2284bd-kserve-59bb44d64c-nvkhk\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.466573 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.466526    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 13:59:51.740965 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.740940    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"]
Sep 16 13:59:51.743428 ip-10-0-133-251 kubenswrapper[2556]: W0916 13:59:51.743392    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf4c0f3_9b28_43a6_9990_fc5e665b79c4.slice/crio-9956e24de714d22c4b25d2b08d06f20364e0065308f46d74cdd1f8f7e5e495a0 WatchSource:0}: Error finding container 9956e24de714d22c4b25d2b08d06f20364e0065308f46d74cdd1f8f7e5e495a0: Status 404 returned error can't find the container with id 9956e24de714d22c4b25d2b08d06f20364e0065308f46d74cdd1f8f7e5e495a0
Sep 16 13:59:51.855732 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.855700    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk" event={"ID":"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4","Type":"ContainerStarted","Data":"551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3"}
Sep 16 13:59:51.855881 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.855744    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk" event={"ID":"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4","Type":"ContainerStarted","Data":"9956e24de714d22c4b25d2b08d06f20364e0065308f46d74cdd1f8f7e5e495a0"}
Sep 16 13:59:51.873511 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:51.873448    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk" podStartSLOduration=0.873433793 podStartE2EDuration="873.433793ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:59:51.871838522 +0000 UTC m=+1602.337775428" watchObservedRunningTime="2026-09-16 13:59:51.873433793 +0000 UTC m=+1602.339370697"
Sep 16 13:59:59.628083 ip-10-0-133-251 kubenswrapper[2556]: I0916 13:59:59.628047    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:00:00.134884 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.134844    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"]
Sep 16 14:00:00.138485 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.138467    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"
Sep 16 14:00:00.140403 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.140379    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zmnnx\""
Sep 16 14:00:00.148493 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.148469    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"]
Sep 16 14:00:00.151688 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.151671    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"
Sep 16 14:00:00.425792 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.425763    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"]
Sep 16 14:00:00.427693 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:00:00.427650    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2c2a83_9ea8_4d11_abc5_bb95dcd72da8.slice/crio-75092cdd8013f7f2af1f7c0819081478c8045043d097f046de436a3eb7f8c87d WatchSource:0}: Error finding container 75092cdd8013f7f2af1f7c0819081478c8045043d097f046de436a3eb7f8c87d: Status 404 returned error can't find the container with id 75092cdd8013f7f2af1f7c0819081478c8045043d097f046de436a3eb7f8c87d
Sep 16 14:00:00.917387 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:00.917359    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb" event={"ID":"af2c2a83-9ea8-4d11-abc5-bb95dcd72da8","Type":"ContainerStarted","Data":"75092cdd8013f7f2af1f7c0819081478c8045043d097f046de436a3eb7f8c87d"}
Sep 16 14:00:01.467473 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:01.467438    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 14:00:01.467650 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:01.467494    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 14:00:01.479764 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:01.479735    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 14:00:01.935387 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:01.935352    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 14:00:04.945481 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:04.945442    2556 generic.go:356] "Generic (PLEG): container finished" podID="af2c2a83-9ea8-4d11-abc5-bb95dcd72da8" containerID="62ce0c59c4f25ba0500e002b32263d95d42ab0b9b4e0c710df9099d39506ce80" exitCode=0
Sep 16 14:00:04.945859 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:04.945489    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb" event={"ID":"af2c2a83-9ea8-4d11-abc5-bb95dcd72da8","Type":"ContainerDied","Data":"62ce0c59c4f25ba0500e002b32263d95d42ab0b9b4e0c710df9099d39506ce80"}
Sep 16 14:00:06.092817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:06.092785    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"
Sep 16 14:00:06.958379 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:06.958349    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb"
Sep 16 14:00:06.958546 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:06.958343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826120-rkssb" event={"ID":"af2c2a83-9ea8-4d11-abc5-bb95dcd72da8","Type":"ContainerDied","Data":"75092cdd8013f7f2af1f7c0819081478c8045043d097f046de436a3eb7f8c87d"}
Sep 16 14:00:06.958546 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:06.958475    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75092cdd8013f7f2af1f7c0819081478c8045043d097f046de436a3eb7f8c87d"
Sep 16 14:00:18.847476 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.847443    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"]
Sep 16 14:00:18.847858 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.847842    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af2c2a83-9ea8-4d11-abc5-bb95dcd72da8" containerName="cleanup"
Sep 16 14:00:18.847858 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.847852    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="af2c2a83-9ea8-4d11-abc5-bb95dcd72da8" containerName="cleanup"
Sep 16 14:00:18.847957 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.847947    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="af2c2a83-9ea8-4d11-abc5-bb95dcd72da8" containerName="cleanup"
Sep 16 14:00:18.852690 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.852674    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:18.855415 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.855391    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"default-dockercfg-tbm2p\""
Sep 16 14:00:18.855600 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.855578    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"openshift-service-ca.crt\""
Sep 16 14:00:18.855682 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.855591    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"kube-root-ca.crt\""
Sep 16 14:00:18.855733 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.855591    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"sub-test-model-ac8045-kserve-self-signed-certs\""
Sep 16 14:00:18.861407 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.861384    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"]
Sep 16 14:00:18.937238 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.937201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:18.966236 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.966205    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-tmp-dir\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:18.966383 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.966261    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-home\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:18.966383 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.966318    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bwj\" (UniqueName: \"kubernetes.io/projected/ff22b844-f166-4db9-8632-e887e5c45c8a-kube-api-access-r7bwj\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:18.966383 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.966339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-model-cache\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:18.966383 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.966373    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-dshm\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:18.966527 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:18.966446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22b844-f166-4db9-8632-e887e5c45c8a-tls-certs\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067539 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-home\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067711 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bwj\" (UniqueName: \"kubernetes.io/projected/ff22b844-f166-4db9-8632-e887e5c45c8a-kube-api-access-r7bwj\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067711 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-model-cache\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067800 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067735    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-dshm\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067800 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22b844-f166-4db9-8632-e887e5c45c8a-tls-certs\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067913 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067829    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-tmp-dir\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.067975 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.067909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-home\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.068156 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.068134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-model-cache\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.068248 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.068233    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-tmp-dir\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.070198 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.070183    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-dshm\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.070403 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.070386    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22b844-f166-4db9-8632-e887e5c45c8a-tls-certs\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.075907 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.075885    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bwj\" (UniqueName: \"kubernetes.io/projected/ff22b844-f166-4db9-8632-e887e5c45c8a-kube-api-access-r7bwj\") pod \"sub-test-model-ac8045-kserve-f84c7497-t5lb8\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.162973 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.162874    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:19.435148 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:19.435121    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"]
Sep 16 14:00:19.437216 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:00:19.437185    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff22b844_f166_4db9_8632_e887e5c45c8a.slice/crio-991467ff0fd677fc98482109e007fd30210efc1a2b8edf17f2bbdde6fad9d663 WatchSource:0}: Error finding container 991467ff0fd677fc98482109e007fd30210efc1a2b8edf17f2bbdde6fad9d663: Status 404 returned error can't find the container with id 991467ff0fd677fc98482109e007fd30210efc1a2b8edf17f2bbdde6fad9d663
Sep 16 14:00:20.033884 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:20.033852    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8" event={"ID":"ff22b844-f166-4db9-8632-e887e5c45c8a","Type":"ContainerStarted","Data":"8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7"}
Sep 16 14:00:20.033884 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:20.033890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8" event={"ID":"ff22b844-f166-4db9-8632-e887e5c45c8a","Type":"ContainerStarted","Data":"991467ff0fd677fc98482109e007fd30210efc1a2b8edf17f2bbdde6fad9d663"}
Sep 16 14:00:20.050960 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:20.050905    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8" podStartSLOduration=2.050889336 podStartE2EDuration="2.050889336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:20.049960074 +0000 UTC m=+1630.515896984" watchObservedRunningTime="2026-09-16 14:00:20.050889336 +0000 UTC m=+1630.516826404"
Sep 16 14:00:22.636099 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:22.636048    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:00:29.162982 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:29.162937    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:29.163477 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:29.163000    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:29.175946 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:29.175920    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:30.105023 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:30.104988    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:00:39.445062 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:39.445024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:00:44.408482 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:44.408435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:44.708698 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:44.708662    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:00:52.133890 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:52.133854    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:00:56.197762 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.197720    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7695-e27038"]
Sep 16 14:00:56.200477 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.200455    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:00:56.202399 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.202379    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cmn8d\""
Sep 16 14:00:56.205966 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.205944    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-e27038"]
Sep 16 14:00:56.328303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.328269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7x6\" (UniqueName: \"kubernetes.io/projected/8ac29d80-924f-435c-bb83-bff978780484-kube-api-access-lg7x6\") pod \"test-curl-7695-e27038\" (UID: \"8ac29d80-924f-435c-bb83-bff978780484\") " pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:00:56.429211 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.429177    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7x6\" (UniqueName: \"kubernetes.io/projected/8ac29d80-924f-435c-bb83-bff978780484-kube-api-access-lg7x6\") pod \"test-curl-7695-e27038\" (UID: \"8ac29d80-924f-435c-bb83-bff978780484\") " pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:00:56.437105 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.437067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7x6\" (UniqueName: \"kubernetes.io/projected/8ac29d80-924f-435c-bb83-bff978780484-kube-api-access-lg7x6\") pod \"test-curl-7695-e27038\" (UID: \"8ac29d80-924f-435c-bb83-bff978780484\") " pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:00:56.512245 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.512151    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:00:56.782932 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:56.782900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7695-e27038"]
Sep 16 14:00:57.253005 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:57.252968    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-e27038" event={"ID":"8ac29d80-924f-435c-bb83-bff978780484","Type":"ContainerStarted","Data":"717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa"}
Sep 16 14:00:57.253436 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:57.253012    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7695-e27038" event={"ID":"8ac29d80-924f-435c-bb83-bff978780484","Type":"ContainerStarted","Data":"ca3bff4febc698c289105b64bf00a551ef405094059140793e8bd6742f65871d"}
Sep 16 14:00:57.268703 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:57.268640    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7695-e27038" podStartSLOduration=1.26862299 podStartE2EDuration="1.26862299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:00:57.26676404 +0000 UTC m=+1667.732700949" watchObservedRunningTime="2026-09-16 14:00:57.26862299 +0000 UTC m=+1667.734559928"
Sep 16 14:00:57.844668 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:57.844633    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7695-e27038"]
Sep 16 14:00:57.848172 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:57.848135    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7695-e27038"]
Sep 16 14:00:58.259503 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:00:58.259471    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7695-e27038" container="test-curl-7695-e27038"
Sep 16 14:00:59.264965 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:59.264905    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7695-e27038" podUID="8ac29d80-924f-435c-bb83-bff978780484" containerName="test-curl-7695-e27038" containerID="cri-o://717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa" gracePeriod=2
Sep 16 14:00:59.892835 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:59.892796    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"]
Sep 16 14:00:59.916233 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:00:59.916202    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:00.861393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:00.861361    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"]
Sep 16 14:01:00.897030 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:00.897002    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:01.411268 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:01.411241    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:01:01.577242 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:01.577145    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg7x6\" (UniqueName: \"kubernetes.io/projected/8ac29d80-924f-435c-bb83-bff978780484-kube-api-access-lg7x6\") pod \"8ac29d80-924f-435c-bb83-bff978780484\" (UID: \"8ac29d80-924f-435c-bb83-bff978780484\") "
Sep 16 14:01:01.579508 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:01.579482    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac29d80-924f-435c-bb83-bff978780484-kube-api-access-lg7x6" (OuterVolumeSpecName: "kube-api-access-lg7x6") pod "8ac29d80-924f-435c-bb83-bff978780484" (UID: "8ac29d80-924f-435c-bb83-bff978780484"). InnerVolumeSpecName "kube-api-access-lg7x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:01.678699 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:01.678662    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg7x6\" (UniqueName: \"kubernetes.io/projected/8ac29d80-924f-435c-bb83-bff978780484-kube-api-access-lg7x6\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:02.059235 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:02.059197    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac29d80-924f-435c-bb83-bff978780484" path="/var/lib/kubelet/pods/8ac29d80-924f-435c-bb83-bff978780484/volumes"
Sep 16 14:01:02.285580 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:02.285545    2556 generic.go:356] "Generic (PLEG): container finished" podID="8ac29d80-924f-435c-bb83-bff978780484" containerID="717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa" exitCode=137
Sep 16 14:01:02.285773 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:02.285586    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7695-e27038"
Sep 16 14:01:02.285773 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:02.285640    2556 scope.go:117] "RemoveContainer" containerID="717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa"
Sep 16 14:01:02.295132 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:02.295111    2556 scope.go:117] "RemoveContainer" containerID="717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa"
Sep 16 14:01:02.295425 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:01:02.295408    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa\": container with ID starting with 717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa not found: ID does not exist" containerID="717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa"
Sep 16 14:01:02.295490 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:02.295434    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa"} err="failed to get container status \"717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa\": rpc error: code = NotFound desc = could not find container \"717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa\": container with ID starting with 717b5dbe00ef324db1a1c3e059fc7021f59c976ca601987b99cb7b4adb9d1dfa not found: ID does not exist"
Sep 16 14:01:03.128462 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:03.128414    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:01:06.978429 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:06.978368    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:07.039592 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:07.039565    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:10.152206 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.152170    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"]
Sep 16 14:01:10.152809 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.152793    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac29d80-924f-435c-bb83-bff978780484" containerName="test-curl-7695-e27038"
Sep 16 14:01:10.152875 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.152812    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac29d80-924f-435c-bb83-bff978780484" containerName="test-curl-7695-e27038"
Sep 16 14:01:10.152950 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.152939    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac29d80-924f-435c-bb83-bff978780484" containerName="test-curl-7695-e27038"
Sep 16 14:01:10.156020 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.156004    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.157971 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.157940    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2284bd\"/\"rate-test-model-2284bd-kserve-self-signed-certs\""
Sep 16 14:01:10.164297 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.164272    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"]
Sep 16 14:01:10.173848 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.173820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:10.253053 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.253000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-dshm\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.253053 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.253054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-tmp-dir\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.253272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.253163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-model-cache\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.253272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.253200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b442bfe-40e1-4a53-975a-878955ee0957-tls-certs\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.253272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.253240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6cs5\" (UniqueName: \"kubernetes.io/projected/2b442bfe-40e1-4a53-975a-878955ee0957-kube-api-access-v6cs5\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.253443 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.253302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-home\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.354816 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.354786    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-home\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355046 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.354882    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-dshm\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355046 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.354905    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-tmp-dir\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355046 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.354946    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-model-cache\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355248 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.355072    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b442bfe-40e1-4a53-975a-878955ee0957-tls-certs\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355248 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.355159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6cs5\" (UniqueName: \"kubernetes.io/projected/2b442bfe-40e1-4a53-975a-878955ee0957-kube-api-access-v6cs5\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355356 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.355329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-home\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355419 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.355351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-model-cache\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.355776 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.355738    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-tmp-dir\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.358460 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.358437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-dshm\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.358697 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.358681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b442bfe-40e1-4a53-975a-878955ee0957-tls-certs\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.362275 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.362253    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6cs5\" (UniqueName: \"kubernetes.io/projected/2b442bfe-40e1-4a53-975a-878955ee0957-kube-api-access-v6cs5\") pod \"rate-test-model-2284bd-kserve-bd7455579-bhpdk\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") " pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.466792 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.466756    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:10.742966 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:10.742934    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"]
Sep 16 14:01:10.743618 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:01:10.743594    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b442bfe_40e1_4a53_975a_878955ee0957.slice/crio-d69c3f10fb13a428ad662a14e5ed0958a0f44922f7aca4470bccdf86a08606b8 WatchSource:0}: Error finding container d69c3f10fb13a428ad662a14e5ed0958a0f44922f7aca4470bccdf86a08606b8: Status 404 returned error can't find the container with id d69c3f10fb13a428ad662a14e5ed0958a0f44922f7aca4470bccdf86a08606b8
Sep 16 14:01:11.343445 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:11.343409    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk" event={"ID":"2b442bfe-40e1-4a53-975a-878955ee0957","Type":"ContainerStarted","Data":"460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b"}
Sep 16 14:01:11.343858 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:11.343453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk" event={"ID":"2b442bfe-40e1-4a53-975a-878955ee0957","Type":"ContainerStarted","Data":"d69c3f10fb13a428ad662a14e5ed0958a0f44922f7aca4470bccdf86a08606b8"}
Sep 16 14:01:11.360452 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:11.360404    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk" podStartSLOduration=1.360387738 podStartE2EDuration="1.360387738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:11.358363061 +0000 UTC m=+1681.824300002" watchObservedRunningTime="2026-09-16 14:01:11.360387738 +0000 UTC m=+1681.826324650"
Sep 16 14:01:14.913981 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:14.913915    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk" podUID="2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" containerName="main" containerID="cri-o://551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3" gracePeriod=45
Sep 16 14:01:15.164023 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.163960    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 14:01:15.205825 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.205799    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-dshm\") pod \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") "
Sep 16 14:01:15.205991 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.205861    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tls-certs\") pod \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") "
Sep 16 14:01:15.205991 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.205908    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tmp-dir\") pod \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") "
Sep 16 14:01:15.205991 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.205932    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-home\") pod \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") "
Sep 16 14:01:15.205991 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.205963    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhvzq\" (UniqueName: \"kubernetes.io/projected/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-kube-api-access-zhvzq\") pod \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") "
Sep 16 14:01:15.206230 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.206004    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-model-cache\") pod \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\" (UID: \"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4\") "
Sep 16 14:01:15.206230 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.206150    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" (UID: "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:15.206230 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.206210    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-home" (OuterVolumeSpecName: "home") pod "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" (UID: "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:15.206390 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.206366    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" (UID: "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:15.206678 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.206461    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:15.206678 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.206481    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:15.208305 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.208278    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" (UID: "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:01:15.208305 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.208297    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-kube-api-access-zhvzq" (OuterVolumeSpecName: "kube-api-access-zhvzq") pod "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" (UID: "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4"). InnerVolumeSpecName "kube-api-access-zhvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:15.208428 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.208345    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-dshm" (OuterVolumeSpecName: "dshm") pod "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" (UID: "2cf4c0f3-9b28-43a6-9990-fc5e665b79c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:15.308025 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.307983    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:15.308224 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.308032    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvzq\" (UniqueName: \"kubernetes.io/projected/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-kube-api-access-zhvzq\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:15.308224 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.308048    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:15.308224 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.308059    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:15.338995 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.338962    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:01:15.367871 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.367838    2556 generic.go:356] "Generic (PLEG): container finished" podID="2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" containerID="551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3" exitCode=0
Sep 16 14:01:15.368021 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.367886    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk" event={"ID":"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4","Type":"ContainerDied","Data":"551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3"}
Sep 16 14:01:15.368021 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.367886    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"
Sep 16 14:01:15.368021 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.367912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk" event={"ID":"2cf4c0f3-9b28-43a6-9990-fc5e665b79c4","Type":"ContainerDied","Data":"9956e24de714d22c4b25d2b08d06f20364e0065308f46d74cdd1f8f7e5e495a0"}
Sep 16 14:01:15.368021 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.367928    2556 scope.go:117] "RemoveContainer" containerID="551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3"
Sep 16 14:01:15.377919 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.377902    2556 scope.go:117] "RemoveContainer" containerID="551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3"
Sep 16 14:01:15.378219 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:01:15.378200    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3\": container with ID starting with 551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3 not found: ID does not exist" containerID="551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3"
Sep 16 14:01:15.378300 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.378239    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3"} err="failed to get container status \"551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3\": rpc error: code = NotFound desc = could not find container \"551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3\": container with ID starting with 551022e01d0c42f9644747afde37c5510da1bc39fe1b494dde8e6815bd63a6b3 not found: ID does not exist"
Sep 16 14:01:15.389221 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.389197    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"]
Sep 16 14:01:15.394859 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.394828    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/sub-test-model-2284bd-kserve-59bb44d64c-nvkhk"]
Sep 16 14:01:15.883250 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:15.883212    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8" podUID="ff22b844-f166-4db9-8632-e887e5c45c8a" containerName="main" containerID="cri-o://8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7" gracePeriod=45
Sep 16 14:01:16.060025 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.059994    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" path="/var/lib/kubelet/pods/2cf4c0f3-9b28-43a6-9990-fc5e665b79c4/volumes"
Sep 16 14:01:16.146520 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.146434    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:01:16.216399 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216364    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7bwj\" (UniqueName: \"kubernetes.io/projected/ff22b844-f166-4db9-8632-e887e5c45c8a-kube-api-access-r7bwj\") pod \"ff22b844-f166-4db9-8632-e887e5c45c8a\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") "
Sep 16 14:01:16.216556 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216433    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-tmp-dir\") pod \"ff22b844-f166-4db9-8632-e887e5c45c8a\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") "
Sep 16 14:01:16.216556 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216459    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22b844-f166-4db9-8632-e887e5c45c8a-tls-certs\") pod \"ff22b844-f166-4db9-8632-e887e5c45c8a\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") "
Sep 16 14:01:16.216664 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216639    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-dshm\") pod \"ff22b844-f166-4db9-8632-e887e5c45c8a\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") "
Sep 16 14:01:16.216705 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216670    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff22b844-f166-4db9-8632-e887e5c45c8a" (UID: "ff22b844-f166-4db9-8632-e887e5c45c8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:16.216705 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216682    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-home\") pod \"ff22b844-f166-4db9-8632-e887e5c45c8a\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") "
Sep 16 14:01:16.216785 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216707    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-model-cache\") pod \"ff22b844-f166-4db9-8632-e887e5c45c8a\" (UID: \"ff22b844-f166-4db9-8632-e887e5c45c8a\") "
Sep 16 14:01:16.216927 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216906    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-home" (OuterVolumeSpecName: "home") pod "ff22b844-f166-4db9-8632-e887e5c45c8a" (UID: "ff22b844-f166-4db9-8632-e887e5c45c8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:16.217010 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.216987    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff22b844-f166-4db9-8632-e887e5c45c8a" (UID: "ff22b844-f166-4db9-8632-e887e5c45c8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:16.217179 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.217163    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:16.217251 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.217184    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:16.217251 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.217199    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:16.218906 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.218882    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff22b844-f166-4db9-8632-e887e5c45c8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff22b844-f166-4db9-8632-e887e5c45c8a" (UID: "ff22b844-f166-4db9-8632-e887e5c45c8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:01:16.219395 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.219370    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-dshm" (OuterVolumeSpecName: "dshm") pod "ff22b844-f166-4db9-8632-e887e5c45c8a" (UID: "ff22b844-f166-4db9-8632-e887e5c45c8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:01:16.219485 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.219399    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff22b844-f166-4db9-8632-e887e5c45c8a-kube-api-access-r7bwj" (OuterVolumeSpecName: "kube-api-access-r7bwj") pod "ff22b844-f166-4db9-8632-e887e5c45c8a" (UID: "ff22b844-f166-4db9-8632-e887e5c45c8a"). InnerVolumeSpecName "kube-api-access-r7bwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:01:16.317851 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.317819    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7bwj\" (UniqueName: \"kubernetes.io/projected/ff22b844-f166-4db9-8632-e887e5c45c8a-kube-api-access-r7bwj\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:16.317851 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.317845    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff22b844-f166-4db9-8632-e887e5c45c8a-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:16.317851 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.317854    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff22b844-f166-4db9-8632-e887e5c45c8a-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:01:16.376796 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.376767    2556 generic.go:356] "Generic (PLEG): container finished" podID="ff22b844-f166-4db9-8632-e887e5c45c8a" containerID="8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7" exitCode=0
Sep 16 14:01:16.376969 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.376811    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"
Sep 16 14:01:16.376969 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.376857    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8" event={"ID":"ff22b844-f166-4db9-8632-e887e5c45c8a","Type":"ContainerDied","Data":"8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7"}
Sep 16 14:01:16.376969 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.376902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8" event={"ID":"ff22b844-f166-4db9-8632-e887e5c45c8a","Type":"ContainerDied","Data":"991467ff0fd677fc98482109e007fd30210efc1a2b8edf17f2bbdde6fad9d663"}
Sep 16 14:01:16.376969 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.376922    2556 scope.go:117] "RemoveContainer" containerID="8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7"
Sep 16 14:01:16.386703 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.386683    2556 scope.go:117] "RemoveContainer" containerID="8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7"
Sep 16 14:01:16.386967 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:01:16.386947    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7\": container with ID starting with 8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7 not found: ID does not exist" containerID="8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7"
Sep 16 14:01:16.387025 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.386975    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7"} err="failed to get container status \"8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7\": rpc error: code = NotFound desc = could not find container \"8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7\": container with ID starting with 8cea9c6e8145133a9bc817c77fe39eb56e8b4183f4dc564939d242592733d5a7 not found: ID does not exist"
Sep 16 14:01:16.397622 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.397570    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"]
Sep 16 14:01:16.405487 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:16.405459    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/sub-test-model-ac8045-kserve-f84c7497-t5lb8"]
Sep 16 14:01:18.059933 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:18.059896    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff22b844-f166-4db9-8632-e887e5c45c8a" path="/var/lib/kubelet/pods/ff22b844-f166-4db9-8632-e887e5c45c8a/volumes"
Sep 16 14:01:20.467012 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:20.466973    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:20.467012 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:20.467019    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:20.479954 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:20.479927    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:21.417872 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:21.417842    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:01:26.612528 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:26.612456    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:35.048220 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048143    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"]
Sep 16 14:01:35.048654 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048561    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff22b844-f166-4db9-8632-e887e5c45c8a" containerName="main"
Sep 16 14:01:35.048654 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048571    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff22b844-f166-4db9-8632-e887e5c45c8a" containerName="main"
Sep 16 14:01:35.048654 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048584    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" containerName="main"
Sep 16 14:01:35.048654 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048590    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" containerName="main"
Sep 16 14:01:35.048787 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048658    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff22b844-f166-4db9-8632-e887e5c45c8a" containerName="main"
Sep 16 14:01:35.048787 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.048670    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf4c0f3-9b28-43a6-9990-fc5e665b79c4" containerName="main"
Sep 16 14:01:35.053397 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.053368    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.056287 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.056260    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"rate-test-model-ac8045-kserve-self-signed-certs\""
Sep 16 14:01:35.056702 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.056264    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"default-dockercfg-tbm2p\""
Sep 16 14:01:35.056702 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.056408    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"openshift-service-ca.crt\""
Sep 16 14:01:35.056702 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.056510    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ac8045\"/\"kube-root-ca.crt\""
Sep 16 14:01:35.060771 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.060746    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"]
Sep 16 14:01:35.095314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.095291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:01:35.208537 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.208505    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-model-cache\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.208727 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.208577    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-dshm\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.208727 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.208630    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-home\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.208727 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.208656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tmp-dir\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.208727 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.208685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nsf4\" (UniqueName: \"kubernetes.io/projected/4c2e808f-7ec1-4473-adf7-764c6c7412a2-kube-api-access-4nsf4\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.208920 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.208744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tls-certs\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.309985 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.309868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-model-cache\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.309985 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.309963    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-dshm\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310261 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-home\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310261 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tmp-dir\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310261 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nsf4\" (UniqueName: \"kubernetes.io/projected/4c2e808f-7ec1-4473-adf7-764c6c7412a2-kube-api-access-4nsf4\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310261 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310157    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tls-certs\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310481 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310287    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-model-cache\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310481 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310436    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tmp-dir\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.310481 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.310439    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-home\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.312418 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.312387    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-dshm\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.312715 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.312699    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tls-certs\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.317943 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.317912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nsf4\" (UniqueName: \"kubernetes.io/projected/4c2e808f-7ec1-4473-adf7-764c6c7412a2-kube-api-access-4nsf4\") pod \"rate-test-model-ac8045-kserve-c58b94687-znm98\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") " pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.365661 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.365619    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:35.638169 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:35.638143    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"]
Sep 16 14:01:35.639735 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:01:35.639706    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c2e808f_7ec1_4473_adf7_764c6c7412a2.slice/crio-dd952f2e79f0d2947abd42e0e41ab3bdbb55fa929d3c44b0ea072f3db476846a WatchSource:0}: Error finding container dd952f2e79f0d2947abd42e0e41ab3bdbb55fa929d3c44b0ea072f3db476846a: Status 404 returned error can't find the container with id dd952f2e79f0d2947abd42e0e41ab3bdbb55fa929d3c44b0ea072f3db476846a
Sep 16 14:01:36.502496 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:36.502451    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98" event={"ID":"4c2e808f-7ec1-4473-adf7-764c6c7412a2","Type":"ContainerStarted","Data":"97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b"}
Sep 16 14:01:36.502496 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:36.502500    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98" event={"ID":"4c2e808f-7ec1-4473-adf7-764c6c7412a2","Type":"ContainerStarted","Data":"dd952f2e79f0d2947abd42e0e41ab3bdbb55fa929d3c44b0ea072f3db476846a"}
Sep 16 14:01:36.518770 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:36.518721    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98" podStartSLOduration=1.518703688 podStartE2EDuration="1.518703688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:01:36.516573331 +0000 UTC m=+1706.982510244" watchObservedRunningTime="2026-09-16 14:01:36.518703688 +0000 UTC m=+1706.984640616"
Sep 16 14:01:38.553564 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:38.553530    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:01:45.365758 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:45.365720    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:45.366199 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:45.365808    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:45.378900 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:45.378872    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:45.568438 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:45.568411    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:01:54.229979 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:01:54.229943    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:02:04.029167 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:04.029126    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:02:09.971633 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:09.971598    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"]
Sep 16 14:02:09.988145 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:09.988065    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:10.946898 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:10.946853    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"]
Sep 16 14:02:10.965734 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:10.965694    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:11.456689 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.456635    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:11.746249 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.746170    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"]
Sep 16 14:02:11.749830 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.749810    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.751932 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.751908    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2ed470-openshift-default-dockercfg-q8kn7\""
Sep 16 14:02:11.762230 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.762204    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"]
Sep 16 14:02:11.774634 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.774611    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:11.860526 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-credential-socket\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860526 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860530    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-token\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860744 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-data\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860744 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860652    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwzqs\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-kube-api-access-kwzqs\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860744 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-certs\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860843 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860747    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-envoy\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860843 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e330328-2c96-47ef-a32c-9d41b97ca03c-istiod-ca-cert\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860843 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-podinfo\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.860940 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.860867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-socket\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962308 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-envoy\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962308 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962308    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e330328-2c96-47ef-a32c-9d41b97ca03c-istiod-ca-cert\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962584 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962434    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-podinfo\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962584 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962506    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-socket\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962584 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962541    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-credential-socket\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962584 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962568    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-token\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962806 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962648    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-data\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962806 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962676    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwzqs\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-kube-api-access-kwzqs\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962806 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962745    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-certs\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962970 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-socket\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.962970 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.962890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-credential-socket\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.963112 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.963039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e330328-2c96-47ef-a32c-9d41b97ca03c-istiod-ca-cert\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.963112 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.963049    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-certs\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.963112 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.963115    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-data\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.964859 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.964839    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-envoy\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.965343 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.965327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-podinfo\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.970437 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.970409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-token\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:11.970791 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:11.970769    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwzqs\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-kube-api-access-kwzqs\") pod \"e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") " pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:12.022619 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.022531    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:12.061620 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.061592    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:12.341687 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.341661    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"]
Sep 16 14:02:12.343980 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:02:12.343950    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e330328_2c96_47ef_a32c_9d41b97ca03c.slice/crio-f239bdbec39f6435f49bd163588bf933c41178de32e71ba3d36f5ce9ee1e81ae WatchSource:0}: Error finding container f239bdbec39f6435f49bd163588bf933c41178de32e71ba3d36f5ce9ee1e81ae: Status 404 returned error can't find the container with id f239bdbec39f6435f49bd163588bf933c41178de32e71ba3d36f5ce9ee1e81ae
Sep 16 14:02:12.346313 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.346278    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:12.346427 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.346361    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:12.346427 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.346403    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:12.731320 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.731281    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" event={"ID":"7e330328-2c96-47ef-a32c-9d41b97ca03c","Type":"ContainerStarted","Data":"a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"}
Sep 16 14:02:12.731819 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.731356    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" event={"ID":"7e330328-2c96-47ef-a32c-9d41b97ca03c","Type":"ContainerStarted","Data":"f239bdbec39f6435f49bd163588bf933c41178de32e71ba3d36f5ce9ee1e81ae"}
Sep 16 14:02:12.752545 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:12.752496    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podStartSLOduration=1.75248246 podStartE2EDuration="1.75248246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:12.750291033 +0000 UTC m=+1743.216227940" watchObservedRunningTime="2026-09-16 14:02:12.75248246 +0000 UTC m=+1743.218419368"
Sep 16 14:02:13.062635 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:13.062531    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:13.068007 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:13.067982    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:13.737350 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:13.737315    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:13.738453 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:13.738431    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:13.987776 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:13.987692    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:14.531035 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:14.530994    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:02:14.755272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:14.755239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:14.856259 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:14.856168    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:15.516067 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:15.516031    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:16.387425 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:16.387395    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:16.492734 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:16.492701    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:16.587218 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:16.587178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:16.844830 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:16.844775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:16.950263 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:16.950229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:25.003122 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.002009    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk" podUID="2b442bfe-40e1-4a53-975a-878955ee0957" containerName="main" containerID="cri-o://460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b" gracePeriod=45
Sep 16 14:02:25.265697 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.265639    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:02:25.315133 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.310420    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b442bfe-40e1-4a53-975a-878955ee0957-tls-certs\") pod \"2b442bfe-40e1-4a53-975a-878955ee0957\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") "
Sep 16 14:02:25.315133 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.310477    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-dshm\") pod \"2b442bfe-40e1-4a53-975a-878955ee0957\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") "
Sep 16 14:02:25.315133 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.310541    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-tmp-dir\") pod \"2b442bfe-40e1-4a53-975a-878955ee0957\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") "
Sep 16 14:02:25.315133 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.310577    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-home\") pod \"2b442bfe-40e1-4a53-975a-878955ee0957\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") "
Sep 16 14:02:25.315133 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.310657    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-model-cache\") pod \"2b442bfe-40e1-4a53-975a-878955ee0957\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") "
Sep 16 14:02:25.315133 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.310683    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6cs5\" (UniqueName: \"kubernetes.io/projected/2b442bfe-40e1-4a53-975a-878955ee0957-kube-api-access-v6cs5\") pod \"2b442bfe-40e1-4a53-975a-878955ee0957\" (UID: \"2b442bfe-40e1-4a53-975a-878955ee0957\") "
Sep 16 14:02:25.315610 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.315332    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b442bfe-40e1-4a53-975a-878955ee0957" (UID: "2b442bfe-40e1-4a53-975a-878955ee0957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:25.316018 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.315987    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b442bfe-40e1-4a53-975a-878955ee0957" (UID: "2b442bfe-40e1-4a53-975a-878955ee0957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:25.316304 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.316270    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-home" (OuterVolumeSpecName: "home") pod "2b442bfe-40e1-4a53-975a-878955ee0957" (UID: "2b442bfe-40e1-4a53-975a-878955ee0957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:25.316769 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.316736    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b442bfe-40e1-4a53-975a-878955ee0957-kube-api-access-v6cs5" (OuterVolumeSpecName: "kube-api-access-v6cs5") pod "2b442bfe-40e1-4a53-975a-878955ee0957" (UID: "2b442bfe-40e1-4a53-975a-878955ee0957"). InnerVolumeSpecName "kube-api-access-v6cs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:25.317881 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.317852    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b442bfe-40e1-4a53-975a-878955ee0957-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b442bfe-40e1-4a53-975a-878955ee0957" (UID: "2b442bfe-40e1-4a53-975a-878955ee0957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:02:25.318632 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.318613    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-dshm" (OuterVolumeSpecName: "dshm") pod "2b442bfe-40e1-4a53-975a-878955ee0957" (UID: "2b442bfe-40e1-4a53-975a-878955ee0957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:25.411774 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.411734    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b442bfe-40e1-4a53-975a-878955ee0957-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:25.411774 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.411768    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:25.411774 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.411779    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:25.412027 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.411792    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:25.412027 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.411803    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b442bfe-40e1-4a53-975a-878955ee0957-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:25.412027 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.411814    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6cs5\" (UniqueName: \"kubernetes.io/projected/2b442bfe-40e1-4a53-975a-878955ee0957-kube-api-access-v6cs5\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:25.829042 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.829004    2556 generic.go:356] "Generic (PLEG): container finished" podID="2b442bfe-40e1-4a53-975a-878955ee0957" containerID="460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b" exitCode=0
Sep 16 14:02:25.829272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.829063    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk" event={"ID":"2b442bfe-40e1-4a53-975a-878955ee0957","Type":"ContainerDied","Data":"460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b"}
Sep 16 14:02:25.829272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.829074    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"
Sep 16 14:02:25.829272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.829117    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk" event={"ID":"2b442bfe-40e1-4a53-975a-878955ee0957","Type":"ContainerDied","Data":"d69c3f10fb13a428ad662a14e5ed0958a0f44922f7aca4470bccdf86a08606b8"}
Sep 16 14:02:25.829272 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.829137    2556 scope.go:117] "RemoveContainer" containerID="460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b"
Sep 16 14:02:25.839012 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.838987    2556 scope.go:117] "RemoveContainer" containerID="460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b"
Sep 16 14:02:25.839330 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:02:25.839309    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b\": container with ID starting with 460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b not found: ID does not exist" containerID="460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b"
Sep 16 14:02:25.839395 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.839338    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b"} err="failed to get container status \"460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b\": rpc error: code = NotFound desc = could not find container \"460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b\": container with ID starting with 460cb3acf4ffc319427b82f353ec8bec2386b2b066418508002141995df9933b not found: ID does not exist"
Sep 16 14:02:25.850939 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.850911    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"]
Sep 16 14:02:25.854524 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.854485    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2284bd/rate-test-model-2284bd-kserve-bd7455579-bhpdk"]
Sep 16 14:02:25.968822 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:25.968761    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98" podUID="4c2e808f-7ec1-4473-adf7-764c6c7412a2" containerName="main" containerID="cri-o://97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b" gracePeriod=45
Sep 16 14:02:26.059726 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.059682    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b442bfe-40e1-4a53-975a-878955ee0957" path="/var/lib/kubelet/pods/2b442bfe-40e1-4a53-975a-878955ee0957/volumes"
Sep 16 14:02:26.261969 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.261945    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:02:26.318784 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.318756    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-home\") pod \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") "
Sep 16 14:02:26.318784 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.318789    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nsf4\" (UniqueName: \"kubernetes.io/projected/4c2e808f-7ec1-4473-adf7-764c6c7412a2-kube-api-access-4nsf4\") pod \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") "
Sep 16 14:02:26.319044 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.318851    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tmp-dir\") pod \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") "
Sep 16 14:02:26.319044 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.318877    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tls-certs\") pod \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") "
Sep 16 14:02:26.319044 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.318900    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-model-cache\") pod \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") "
Sep 16 14:02:26.319044 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.319002    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-dshm\") pod \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\" (UID: \"4c2e808f-7ec1-4473-adf7-764c6c7412a2\") "
Sep 16 14:02:26.319291 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.319078    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-home" (OuterVolumeSpecName: "home") pod "4c2e808f-7ec1-4473-adf7-764c6c7412a2" (UID: "4c2e808f-7ec1-4473-adf7-764c6c7412a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:26.319291 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.319166    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c2e808f-7ec1-4473-adf7-764c6c7412a2" (UID: "4c2e808f-7ec1-4473-adf7-764c6c7412a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:26.319399 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.319380    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c2e808f-7ec1-4473-adf7-764c6c7412a2" (UID: "4c2e808f-7ec1-4473-adf7-764c6c7412a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:26.319473 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.319446    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.319533 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.319473    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.321251 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.321223    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-dshm" (OuterVolumeSpecName: "dshm") pod "4c2e808f-7ec1-4473-adf7-764c6c7412a2" (UID: "4c2e808f-7ec1-4473-adf7-764c6c7412a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:02:26.321380 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.321307    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2e808f-7ec1-4473-adf7-764c6c7412a2-kube-api-access-4nsf4" (OuterVolumeSpecName: "kube-api-access-4nsf4") pod "4c2e808f-7ec1-4473-adf7-764c6c7412a2" (UID: "4c2e808f-7ec1-4473-adf7-764c6c7412a2"). InnerVolumeSpecName "kube-api-access-4nsf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:02:26.321380 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.321356    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c2e808f-7ec1-4473-adf7-764c6c7412a2" (UID: "4c2e808f-7ec1-4473-adf7-764c6c7412a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:02:26.419885 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.419853    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nsf4\" (UniqueName: \"kubernetes.io/projected/4c2e808f-7ec1-4473-adf7-764c6c7412a2-kube-api-access-4nsf4\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.419885 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.419882    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c2e808f-7ec1-4473-adf7-764c6c7412a2-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.419885 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.419891    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.420193 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.419899    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c2e808f-7ec1-4473-adf7-764c6c7412a2-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:02:26.635487 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.635450    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:02:26.839155 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.839120    2556 generic.go:356] "Generic (PLEG): container finished" podID="4c2e808f-7ec1-4473-adf7-764c6c7412a2" containerID="97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b" exitCode=0
Sep 16 14:02:26.839340 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.839175    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"
Sep 16 14:02:26.839340 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.839180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98" event={"ID":"4c2e808f-7ec1-4473-adf7-764c6c7412a2","Type":"ContainerDied","Data":"97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b"}
Sep 16 14:02:26.839340 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.839225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98" event={"ID":"4c2e808f-7ec1-4473-adf7-764c6c7412a2","Type":"ContainerDied","Data":"dd952f2e79f0d2947abd42e0e41ab3bdbb55fa929d3c44b0ea072f3db476846a"}
Sep 16 14:02:26.839340 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.839247    2556 scope.go:117] "RemoveContainer" containerID="97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b"
Sep 16 14:02:26.848564 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.848537    2556 scope.go:117] "RemoveContainer" containerID="97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b"
Sep 16 14:02:26.848889 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:02:26.848863    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b\": container with ID starting with 97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b not found: ID does not exist" containerID="97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b"
Sep 16 14:02:26.848997 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.848899    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b"} err="failed to get container status \"97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b\": rpc error: code = NotFound desc = could not find container \"97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b\": container with ID starting with 97346602d5c5505ef165c4bf35ae5ce551460d0762bf8a32ef09d499a600379b not found: ID does not exist"
Sep 16 14:02:26.867964 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.867935    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"]
Sep 16 14:02:26.872192 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:26.872170    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ac8045/rate-test-model-ac8045-kserve-c58b94687-znm98"]
Sep 16 14:02:28.060248 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:28.060215    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c2e808f-7ec1-4473-adf7-764c6c7412a2" path="/var/lib/kubelet/pods/4c2e808f-7ec1-4473-adf7-764c6c7412a2/volumes"
Sep 16 14:02:29.435687 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.435527    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": read tcp 10.133.0.2:36340->10.133.0.91:15021: read: connection reset by peer" start-of-body=
Sep 16 14:02:29.435687 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.435608    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": read tcp 10.133.0.2:36340->10.133.0.91:15021: read: connection reset by peer"
Sep 16 14:02:29.868927 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.868834    2556 generic.go:356] "Generic (PLEG): container finished" podID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerID="a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552" exitCode=0
Sep 16 14:02:29.868927 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.868912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" event={"ID":"7e330328-2c96-47ef-a32c-9d41b97ca03c","Type":"ContainerDied","Data":"a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"}
Sep 16 14:02:29.869397 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.869379    2556 scope.go:117] "RemoveContainer" containerID="a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"
Sep 16 14:02:29.870495 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.870466    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:29.870591 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.870547    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:29.870655 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:29.870595    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:30.877516 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:30.877477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" event={"ID":"7e330328-2c96-47ef-a32c-9d41b97ca03c","Type":"ContainerStarted","Data":"cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a"}
Sep 16 14:02:31.062017 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:31.061986    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:31.062217 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:31.062153    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 16 14:02:31.062217 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:31.062204    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 16 14:02:32.062788 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:32.062754    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 16 14:02:32.063233 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:32.062815    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 16 14:02:33.062178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:33.062138    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 16 14:02:33.062347 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:33.062213    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 16 14:02:34.062525 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:34.062495    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 16 14:02:34.063001 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:34.062553    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 16 14:02:35.062837 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:35.062789    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 16 14:02:35.063296 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:35.062857    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 16 14:02:36.062320 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:36.062284    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Sep 16 14:02:36.062517 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:36.062353    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Sep 16 14:02:37.065846 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:37.065818    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:37.066239 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:37.066081    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:37.066919 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:37.066899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:02:38.304699 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:38.304669    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:38.977380 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:38.977346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:39.749925 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:39.749876    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:39.849918 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:39.849883    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:40.500232 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.500196    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:40.600459 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.600411    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:40.817744 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.816836    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"]
Sep 16 14:02:40.818206 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.818123    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b442bfe-40e1-4a53-975a-878955ee0957" containerName="main"
Sep 16 14:02:40.818206 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.818147    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b442bfe-40e1-4a53-975a-878955ee0957" containerName="main"
Sep 16 14:02:40.818206 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.818197    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c2e808f-7ec1-4473-adf7-764c6c7412a2" containerName="main"
Sep 16 14:02:40.818206 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.818206    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c2e808f-7ec1-4473-adf7-764c6c7412a2" containerName="main"
Sep 16 14:02:40.818458 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.818437    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c2e808f-7ec1-4473-adf7-764c6c7412a2" containerName="main"
Sep 16 14:02:40.818557 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.818476    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b442bfe-40e1-4a53-975a-878955ee0957" containerName="main"
Sep 16 14:02:40.821971 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.821936    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.822604 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.822582    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"]
Sep 16 14:02:40.824610 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.824588    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9e8244-openshift-default-dockercfg-qj2xc\""
Sep 16 14:02:40.829803 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.829738    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:40.859697 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-certs\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.859911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-istio-token\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.859911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859728    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06783f48-4ceb-4601-af37-34da7f2d593c-istiod-ca-cert\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.859911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-data\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.859911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859862    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-socket\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.860103 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-credential-socket\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.860103 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.859978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-envoy\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.860103 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.860002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06783f48-4ceb-4601-af37-34da7f2d593c-istio-podinfo\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.860103 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.860027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzl9\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-kube-api-access-lmzl9\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.960903 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.960863    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06783f48-4ceb-4601-af37-34da7f2d593c-istio-podinfo\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.960914    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzl9\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-kube-api-access-lmzl9\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.960984    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-certs\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-istio-token\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961045    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06783f48-4ceb-4601-af37-34da7f2d593c-istiod-ca-cert\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961070    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-data\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-socket\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961511 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961222    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-credential-socket\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961511 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-envoy\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961511 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961485    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-certs\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961620 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961562    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-data\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961876 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-socket\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961959 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961905    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-credential-socket\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.961959 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.961942    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06783f48-4ceb-4601-af37-34da7f2d593c-istiod-ca-cert\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.964149 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.964126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-envoy\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.964324 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.964306    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06783f48-4ceb-4601-af37-34da7f2d593c-istio-podinfo\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.969010 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.968983    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-istio-token\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:40.969145 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:40.969029    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzl9\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-kube-api-access-lmzl9\") pod \"e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") " pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:41.134751 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.134659    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:41.273785 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.273749    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:41.414120 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.414074    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"]
Sep 16 14:02:41.415084 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:02:41.415061    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06783f48_4ceb_4601_af37_34da7f2d593c.slice/crio-acadb6712d328bfcfa74a795b218b3330df9cca92897b720bd6a9da4033863a2 WatchSource:0}: Error finding container acadb6712d328bfcfa74a795b218b3330df9cca92897b720bd6a9da4033863a2: Status 404 returned error can't find the container with id acadb6712d328bfcfa74a795b218b3330df9cca92897b720bd6a9da4033863a2
Sep 16 14:02:41.417455 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.417421    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:41.417539 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.417493    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:41.417539 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.417523    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:02:41.962636 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.962591    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" event={"ID":"06783f48-4ceb-4601-af37-34da7f2d593c","Type":"ContainerStarted","Data":"b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"}
Sep 16 14:02:41.963041 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.962662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" event={"ID":"06783f48-4ceb-4601-af37-34da7f2d593c","Type":"ContainerStarted","Data":"acadb6712d328bfcfa74a795b218b3330df9cca92897b720bd6a9da4033863a2"}
Sep 16 14:02:41.979790 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:41.979732    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podStartSLOduration=1.97971072 podStartE2EDuration="1.97971072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:41.979642249 +0000 UTC m=+1772.445579184" watchObservedRunningTime="2026-09-16 14:02:41.97971072 +0000 UTC m=+1772.445647625"
Sep 16 14:02:42.135586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:42.135548    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:42.140550 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:42.140523    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:42.969788 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:42.969750    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:42.970727 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:42.970702    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:02:43.199329 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:43.199293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:43.996770 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:43.996735    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:44.091383 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:44.091349    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:44.824480 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:44.824449    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:45.647543 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:45.647508    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:46.370499 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:46.370466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:47.173684 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.173652    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:47.274676 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.274643    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:47.619011 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.618925    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:47.739183 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.739149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:47.840575 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.840540    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:02:47.863745 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.863702    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"]
Sep 16 14:02:47.868619 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.868594    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:47.870861 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.870797    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9e8244-dockercfg-bpgct\""
Sep 16 14:02:47.870861 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.870816    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9e8244\""
Sep 16 14:02:47.876951 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.876922    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"]
Sep 16 14:02:47.934549 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.934513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjbk\" (UniqueName: \"kubernetes.io/projected/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-kube-api-access-sqjbk\") pod \"payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:47.934771 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:47.934583    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:48.036303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:48.036266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:48.036527 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:48.036368    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjbk\" (UniqueName: \"kubernetes.io/projected/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-kube-api-access-sqjbk\") pod \"payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:48.037132 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:48.037079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:48.044925 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:48.044895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjbk\" (UniqueName: \"kubernetes.io/projected/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-kube-api-access-sqjbk\") pod \"payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:48.181238 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:48.181208    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:48.455579 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:48.455553    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"]
Sep 16 14:02:48.456304 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:02:48.456280    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e8556b_2b30_4416_bb5e_f0f4af8790d8.slice/crio-f04f9396d576c6e136103842ff1e86b4a6ce11b5a348c7e7a3ae0239ce1a2034 WatchSource:0}: Error finding container f04f9396d576c6e136103842ff1e86b4a6ce11b5a348c7e7a3ae0239ce1a2034: Status 404 returned error can't find the container with id f04f9396d576c6e136103842ff1e86b4a6ce11b5a348c7e7a3ae0239ce1a2034
Sep 16 14:02:49.008779 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:49.008726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk" event={"ID":"a1e8556b-2b30-4416-bb5e-f0f4af8790d8","Type":"ContainerStarted","Data":"9e9c7a8338a56e72b8e20651f401e58083799092681d8027062075e4db0506e7"}
Sep 16 14:02:49.008980 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:49.008830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk" event={"ID":"a1e8556b-2b30-4416-bb5e-f0f4af8790d8","Type":"ContainerStarted","Data":"f04f9396d576c6e136103842ff1e86b4a6ce11b5a348c7e7a3ae0239ce1a2034"}
Sep 16 14:02:49.008980 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:49.008860    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:02:49.025426 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:49.025367    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk" podStartSLOduration=2.025352068 podStartE2EDuration="2.025352068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:02:49.023405867 +0000 UTC m=+1779.489342785" watchObservedRunningTime="2026-09-16 14:02:49.025352068 +0000 UTC m=+1779.491288975"
Sep 16 14:02:51.543306 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:51.543247    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:02:59.011316 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:59.011267    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 14:02:59.011715 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:02:59.011338    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 14:03:00.018679 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:00.018644    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:03:01.085735 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.085700    2556 generic.go:356] "Generic (PLEG): container finished" podID="06783f48-4ceb-4601-af37-34da7f2d593c" containerID="b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e" exitCode=0
Sep 16 14:03:01.086174 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.085769    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" event={"ID":"06783f48-4ceb-4601-af37-34da7f2d593c","Type":"ContainerDied","Data":"b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"}
Sep 16 14:03:01.086229 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.086207    2556 scope.go:117] "RemoveContainer" containerID="b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"
Sep 16 14:03:01.087412 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.087373    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:03:01.087516 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.087455    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:03:01.087516 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.087489    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:03:01.091845 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:01.091818    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:02.094241 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:02.094202    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" event={"ID":"06783f48-4ceb-4601-af37-34da7f2d593c","Type":"ContainerStarted","Data":"81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c"}
Sep 16 14:03:02.135687 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:02.135652    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:03:02.135931 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:02.135907    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 16 14:03:02.136020 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:02.135980    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 16 14:03:03.135372 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:03.135336    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 16 14:03:03.135770 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:03.135407    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 16 14:03:04.136116 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:04.136054    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 16 14:03:04.136526 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:04.136158    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 16 14:03:05.136029 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:05.135995    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 16 14:03:05.136242 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:05.136049    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 16 14:03:06.071467 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:06.071427    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:06.135993 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:06.135956    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 16 14:03:06.136346 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:06.136313    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 16 14:03:07.135269 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:07.135227    2556 patch_prober.go:28] interesting pod/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 16 14:03:07.135446 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:07.135276    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 16 14:03:08.139195 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:08.139169    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:03:08.139659 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:08.139579    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:03:08.140372 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:08.140351    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:03:08.263981 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:08.263943    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:10.286921 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:10.286871    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:10.347526 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:10.347486    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:10.998339 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:10.998302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:11.649719 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:11.649686    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:13.486219 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:13.486182    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:14.090517 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:14.090476    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:19.327403 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:19.327367    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6j2c_ed380fe8-2bf2-4ad0-968e-341cbb8a3e17/payload-processing/0.log"
Sep 16 14:03:19.446620 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:19.446585    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk_a1e8556b-2b30-4416-bb5e-f0f4af8790d8/payload-processing/0.log"
Sep 16 14:03:22.341016 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.340974    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"]
Sep 16 14:03:22.344787 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.344765    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.346858 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.346837    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2ed470\"/\"kube-root-ca.crt\""
Sep 16 14:03:22.347415 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.347390    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2ed470\"/\"default-dockercfg-6qnx5\""
Sep 16 14:03:22.347553 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.347425    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2ed470\"/\"openshift-service-ca.crt\""
Sep 16 14:03:22.347553 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.347494    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2ed470\"/\"ipp-route-2ed470-kserve-self-signed-certs\""
Sep 16 14:03:22.354271 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.354246    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"]
Sep 16 14:03:22.369251 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.369215    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:22.494581 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.494540    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tls-certs\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.494777 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.494589    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-model-cache\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.494777 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.494723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-dshm\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.494777 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.494742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tmp-dir\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.494777 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.494759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dnz\" (UniqueName: \"kubernetes.io/projected/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-kube-api-access-r7dnz\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.494920 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.494786    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-home\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.595833 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.595741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tls-certs\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.595833 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.595794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-model-cache\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596038 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.595883    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-dshm\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596038 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.595911    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tmp-dir\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596038 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.595937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dnz\" (UniqueName: \"kubernetes.io/projected/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-kube-api-access-r7dnz\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596038 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.595977    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-home\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596365 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.596340    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-model-cache\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596442 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.596367    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-home\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.596501 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.596440    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tmp-dir\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.598520 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.598494    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-dshm\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.598661 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.598647    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tls-certs\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.603629 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.603603    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dnz\" (UniqueName: \"kubernetes.io/projected/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-kube-api-access-r7dnz\") pod \"ipp-route-2ed470-kserve-f6f659cd7-8mcb4\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") " pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.656065 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.656031    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:22.930441 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:22.930409    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"]
Sep 16 14:03:22.931720 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:03:22.931689    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa3324e6_8396_4d29_8bc5_3ef82d7ca6f0.slice/crio-4aefea9c07cbb060ddaa15896b6fc589af7fa418ecf0fb1ce0fa4aad09615419 WatchSource:0}: Error finding container 4aefea9c07cbb060ddaa15896b6fc589af7fa418ecf0fb1ce0fa4aad09615419: Status 404 returned error can't find the container with id 4aefea9c07cbb060ddaa15896b6fc589af7fa418ecf0fb1ce0fa4aad09615419
Sep 16 14:03:23.234101 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:23.233995    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4" event={"ID":"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0","Type":"ContainerStarted","Data":"006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181"}
Sep 16 14:03:23.234101 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:23.234047    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4" event={"ID":"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0","Type":"ContainerStarted","Data":"4aefea9c07cbb060ddaa15896b6fc589af7fa418ecf0fb1ce0fa4aad09615419"}
Sep 16 14:03:23.250846 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:23.250792    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4" podStartSLOduration=1.250775913 podStartE2EDuration="1.250775913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:23.248983589 +0000 UTC m=+1813.714920515" watchObservedRunningTime="2026-09-16 14:03:23.250775913 +0000 UTC m=+1813.716712824"
Sep 16 14:03:25.932941 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:25.932900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:03:32.657048 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:32.657009    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:32.657478 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:32.657079    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:32.669691 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:32.669659    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:33.313512 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:33.313482    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:03:40.542039 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:40.541995    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:03:47.372170 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:47.372138    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6j2c_ed380fe8-2bf2-4ad0-968e-341cbb8a3e17/payload-processing/0.log"
Sep 16 14:03:47.493701 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:47.493672    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk_a1e8556b-2b30-4416-bb5e-f0f4af8790d8/payload-processing/0.log"
Sep 16 14:03:47.772383 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:47.772336    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:48.048123 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.047988    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"]
Sep 16 14:03:48.051942 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.051918    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.054242 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.054215    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-11bfbd-openshift-default-dockercfg-54cp5\""
Sep 16 14:03:48.063006 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.062974    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"]
Sep 16 14:03:48.165592 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165555    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-socket\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.165817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165637    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-credential-socket\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.165817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.165817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165698    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-data\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.165817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-certs\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.165817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165808    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-envoy\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.166138 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-token\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.166138 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165884    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfql\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-kube-api-access-zsfql\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.166138 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.165930    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-podinfo\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.266951 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.266903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267225 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.266966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-data\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267225 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-certs\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267225 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267079    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-envoy\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267225 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-token\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267225 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267200    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfql\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-kube-api-access-zsfql\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267515 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267242    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-podinfo\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267515 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-socket\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267515 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267377    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-credential-socket\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267944 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267824    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-data\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.267944 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267894    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-certs\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.268144 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.267955    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-socket\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.268144 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.268009    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-credential-socket\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.268144 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.268084    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istiod-ca-cert\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.270166 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.270140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-envoy\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.270331 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.270304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-podinfo\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.275367 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.275334    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-token\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.275534 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.275510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfql\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-kube-api-access-zsfql\") pod \"e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") " pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.366493 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.366395    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:48.645818 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.645790    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"]
Sep 16 14:03:48.647083 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:03:48.647048    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f4c01f_7b88_4057_a5e2_c635fc7d621d.slice/crio-6fc9eef288cb14b771190e9685f63d54f59f386e88de2c471e0164f774554d1d WatchSource:0}: Error finding container 6fc9eef288cb14b771190e9685f63d54f59f386e88de2c471e0164f774554d1d: Status 404 returned error can't find the container with id 6fc9eef288cb14b771190e9685f63d54f59f386e88de2c471e0164f774554d1d
Sep 16 14:03:48.649526 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.649494    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:03:48.649615 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.649560    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:03:48.649615 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:48.649593    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:03:49.406744 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:49.406709    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" event={"ID":"e1f4c01f-7b88-4057-a5e2-c635fc7d621d","Type":"ContainerStarted","Data":"5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"}
Sep 16 14:03:49.406928 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:49.406763    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" event={"ID":"e1f4c01f-7b88-4057-a5e2-c635fc7d621d","Type":"ContainerStarted","Data":"6fc9eef288cb14b771190e9685f63d54f59f386e88de2c471e0164f774554d1d"}
Sep 16 14:03:49.427644 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:49.427586    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podStartSLOduration=1.427566528 podStartE2EDuration="1.427566528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:03:49.423804698 +0000 UTC m=+1839.889741606" watchObservedRunningTime="2026-09-16 14:03:49.427566528 +0000 UTC m=+1839.893503436"
Sep 16 14:03:49.831257 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:49.831222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:50.366880 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:50.366836    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:50.373024 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:50.372996    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:50.413651 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:50.413614    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:50.414673 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:50.414656    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:03:50.673431 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:50.673390    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:51.361910 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:51.361874    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:51.461072 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:51.461036    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:52.289387 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:52.289347    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:53.056821 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:53.056787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:53.794160 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:53.794075    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:54.485321 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:54.485287    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:54.541820 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:54.541779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:54.641101 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:54.641066    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:54.945249 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:54.945209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:55.040711 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:55.040674    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:03:59.569281 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:03:59.569243    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:04:10.553232 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:10.553202    2556 generic.go:356] "Generic (PLEG): container finished" podID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerID="5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd" exitCode=0
Sep 16 14:04:10.553675 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:10.553275    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" event={"ID":"e1f4c01f-7b88-4057-a5e2-c635fc7d621d","Type":"ContainerDied","Data":"5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"}
Sep 16 14:04:10.553739 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:10.553724    2556 scope.go:117] "RemoveContainer" containerID="5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"
Sep 16 14:04:10.554955 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:10.554923    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:04:10.555036 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:10.555017    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:04:10.555106 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:10.555074    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 16 14:04:11.561986 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:11.561945    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" event={"ID":"e1f4c01f-7b88-4057-a5e2-c635fc7d621d","Type":"ContainerStarted","Data":"34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464"}
Sep 16 14:04:12.366648 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:12.366607    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:04:12.366890 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:12.366861    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 16 14:04:12.366984 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:12.366938    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:04:13.366964 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:13.366921    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 16 14:04:13.367403 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:13.366998    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:04:14.366837 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:14.366801    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 16 14:04:14.367031 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:14.366875    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:04:15.367545 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:15.367508    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 16 14:04:15.367969 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:15.367573    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:04:16.367756 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:16.367713    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body=
Sep 16 14:04:16.368173 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:16.367784    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused"
Sep 16 14:04:16.409241 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:16.409204    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:16.509636 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:16.509603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:17.367129 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:17.367073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:17.371276 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:17.371250    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:04:17.371648 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:17.371629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:04:17.372255 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:17.372238    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:04:18.227199 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:18.227163    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:18.973503 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:18.973470    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:19.796501 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:19.796468    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:20.470178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:20.470146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:21.247866 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:21.247824    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:21.921658 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:21.921624    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:22.791845 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:22.791807    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:23.369322 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:23.369270    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:23.468555 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:23.468513    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:23.565603 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:23.565572    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:23.635680 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:23.635592    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:23.956152 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:23.956116    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:24.056832 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:24.056798    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:24.681676 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:24.681646    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:25.490354 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:25.490321    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:26.287025 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:26.286993    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:26.970059 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:26.970022    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:27.637464 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:27.637416    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:04:29.837449 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:29.837417    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:29.936173 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:29.936137    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:30.097073 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:30.096992    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"]
Sep 16 14:04:30.097299 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:30.097222    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" containerID="cri-o://34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464" gracePeriod=30
Sep 16 14:04:30.126130 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:30.126079    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:32.373961 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:32.373876    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 16 14:04:32.374350 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:32.373951    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:04:34.487946 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:34.487892    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:35.358551 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.358478    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:04:35.419723 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.419690    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:35.459848 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.459809    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-data\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.459848 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.459849    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-podinfo\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460131 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.459890    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istiod-ca-cert\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460131 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.459940    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-token\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460131 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.459967    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-certs\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460130    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-socket\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460187    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-envoy\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460221    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsfql\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-kube-api-access-zsfql\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460247    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-data" (OuterVolumeSpecName: "istio-data") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:35.460307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460271    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-credential-socket\") pod \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\" (UID: \"e1f4c01f-7b88-4057-a5e2-c635fc7d621d\") "
Sep 16 14:04:35.460538 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460416    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:35.460538 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460448    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:04:35.460645 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460527    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:35.460703 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460644    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:35.460873 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460841    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.460873 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460864    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.461295 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460879    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.461295 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460888    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.461295 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.460897    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.462495 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.462471    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-token" (OuterVolumeSpecName: "istio-token") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:04:35.462634 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.462616    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:04:35.462928 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.462911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:04:35.463073 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.463052    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-kube-api-access-zsfql" (OuterVolumeSpecName: "kube-api-access-zsfql") pod "e1f4c01f-7b88-4057-a5e2-c635fc7d621d" (UID: "e1f4c01f-7b88-4057-a5e2-c635fc7d621d"). InnerVolumeSpecName "kube-api-access-zsfql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:04:35.562408 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.562371    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.562408 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.562403    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.562408 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.562414    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsfql\" (UniqueName: \"kubernetes.io/projected/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-kube-api-access-zsfql\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.562878 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.562424    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1f4c01f-7b88-4057-a5e2-c635fc7d621d-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:04:35.736804 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.736774    2556 generic.go:356] "Generic (PLEG): container finished" podID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerID="34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464" exitCode=0
Sep 16 14:04:35.736990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.736821    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"
Sep 16 14:04:35.736990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.736881    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" event={"ID":"e1f4c01f-7b88-4057-a5e2-c635fc7d621d","Type":"ContainerDied","Data":"34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464"}
Sep 16 14:04:35.736990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.736936    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm" event={"ID":"e1f4c01f-7b88-4057-a5e2-c635fc7d621d","Type":"ContainerDied","Data":"6fc9eef288cb14b771190e9685f63d54f59f386e88de2c471e0164f774554d1d"}
Sep 16 14:04:35.736990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.736960    2556 scope.go:117] "RemoveContainer" containerID="34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464"
Sep 16 14:04:35.747845 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.747824    2556 scope.go:117] "RemoveContainer" containerID="5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"
Sep 16 14:04:35.757355 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.757327    2556 scope.go:117] "RemoveContainer" containerID="34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464"
Sep 16 14:04:35.757709 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:04:35.757683    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464\": container with ID starting with 34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464 not found: ID does not exist" containerID="34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464"
Sep 16 14:04:35.757759 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.757722    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464"} err="failed to get container status \"34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464\": rpc error: code = NotFound desc = could not find container \"34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464\": container with ID starting with 34bf55cbfea2e3fe885c355d83a7927e849ff6b33a19114a1e90617d3298b464 not found: ID does not exist"
Sep 16 14:04:35.757759 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.757745    2556 scope.go:117] "RemoveContainer" containerID="5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"
Sep 16 14:04:35.758029 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:04:35.758009    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd\": container with ID starting with 5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd not found: ID does not exist" containerID="5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"
Sep 16 14:04:35.758078 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.758039    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd"} err="failed to get container status \"5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd\": rpc error: code = NotFound desc = could not find container \"5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd\": container with ID starting with 5e48116816b48fc642865f0b546c07d04acca9b6a442fefa4260e7d3c8ef3edd not found: ID does not exist"
Sep 16 14:04:35.760013 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.759989    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"]
Sep 16 14:04:35.763571 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:35.763547    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-11bfbd-openshift-default-679bb8df6d-mrncm"]
Sep 16 14:04:36.061270 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:36.061186    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" path="/var/lib/kubelet/pods/e1f4c01f-7b88-4057-a5e2-c635fc7d621d/volumes"
Sep 16 14:04:36.171102 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:36.171064    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:36.974317 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:36.974281    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:37.633792 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:37.633759    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:38.295808 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:38.295773    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:38.705203 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:38.705167    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:38.759691 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:38.759658    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:47.438850 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:47.438814    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:04:49.692797 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:49.692753    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:49.790718 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:49.790679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:49.893562 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:49.893527    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:49.952549 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:49.952435    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:50.269463 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:50.269354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:50.368371 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:50.368323    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:51.083902 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:51.083862    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:51.915344 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:51.915307    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:52.614198 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:52.614165    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:04:55.247494 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:55.247457    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"]
Sep 16 14:04:55.297267 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:04:55.297233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:00.145212 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:00.145174    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:05:10.268252 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.268208    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4" podUID="aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" containerName="main" containerID="cri-o://006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181" gracePeriod=45
Sep 16 14:05:10.520037 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.519980    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:05:10.614115 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614058    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-model-cache\") pod \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") "
Sep 16 14:05:10.614115 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tmp-dir\") pod \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") "
Sep 16 14:05:10.614393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614181    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7dnz\" (UniqueName: \"kubernetes.io/projected/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-kube-api-access-r7dnz\") pod \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") "
Sep 16 14:05:10.614393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614217    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-dshm\") pod \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") "
Sep 16 14:05:10.614393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614268    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tls-certs\") pod \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") "
Sep 16 14:05:10.614393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614286    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-home\") pod \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\" (UID: \"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0\") "
Sep 16 14:05:10.614393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614349    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" (UID: "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:10.614393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614369    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" (UID: "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:10.614765 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614566    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:10.614765 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614581    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:10.614877 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.614765    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-home" (OuterVolumeSpecName: "home") pod "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" (UID: "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:10.616953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.616926    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-dshm" (OuterVolumeSpecName: "dshm") pod "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" (UID: "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:10.617110 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.617035    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-kube-api-access-r7dnz" (OuterVolumeSpecName: "kube-api-access-r7dnz") pod "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" (UID: "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0"). InnerVolumeSpecName "kube-api-access-r7dnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:10.617816 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.617795    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" (UID: "aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:05:10.715110 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.715065    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dnz\" (UniqueName: \"kubernetes.io/projected/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-kube-api-access-r7dnz\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:10.715264 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.715121    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:10.715264 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.715132    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:10.715264 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.715140    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:10.958477 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.958447    2556 generic.go:356] "Generic (PLEG): container finished" podID="aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" containerID="006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181" exitCode=0
Sep 16 14:05:10.958686 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.958491    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"
Sep 16 14:05:10.958686 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.958505    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4" event={"ID":"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0","Type":"ContainerDied","Data":"006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181"}
Sep 16 14:05:10.958686 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.958536    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4" event={"ID":"aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0","Type":"ContainerDied","Data":"4aefea9c07cbb060ddaa15896b6fc589af7fa418ecf0fb1ce0fa4aad09615419"}
Sep 16 14:05:10.958686 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.958553    2556 scope.go:117] "RemoveContainer" containerID="006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181"
Sep 16 14:05:10.968823 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.968802    2556 scope.go:117] "RemoveContainer" containerID="006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181"
Sep 16 14:05:10.969071 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:05:10.969052    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181\": container with ID starting with 006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181 not found: ID does not exist" containerID="006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181"
Sep 16 14:05:10.969160 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.969080    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181"} err="failed to get container status \"006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181\": rpc error: code = NotFound desc = could not find container \"006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181\": container with ID starting with 006398ba20e9b03afc50c0e7c9adbffab6d7aef9ba676bad57e513965e793181 not found: ID does not exist"
Sep 16 14:05:10.979773 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.979749    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"]
Sep 16 14:05:10.983191 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:10.983167    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2ed470/ipp-route-2ed470-kserve-f6f659cd7-8mcb4"]
Sep 16 14:05:12.059768 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:12.059736    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" path="/var/lib/kubelet/pods/aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0/volumes"
Sep 16 14:05:31.832475 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:31.832440    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:32.442688 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:32.442655    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:37.901195 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:37.901158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:37.956604 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:37.956568    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:38.177426 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:38.177389    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"]
Sep 16 14:05:38.177694 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:38.177653    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy" containerID="cri-o://cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a" gracePeriod=30
Sep 16 14:05:38.201393 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:38.201363    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:38.695396 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:38.695361    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:39.505318 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:39.505272    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:40.294303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:40.294270    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:40.973218 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:40.973181    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:41.764404 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:41.764365    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:43.437396 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.437326    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:05:43.541963 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.541924    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-token\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.541977    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-socket\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542005    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwzqs\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-kube-api-access-kwzqs\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542039    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e330328-2c96-47ef-a32c-9d41b97ca03c-istiod-ca-cert\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542133    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-envoy\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542178 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542171    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-certs\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542451 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542201    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-credential-socket\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542451 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542224    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-data\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542451 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542286    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:43.542451 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542300    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-podinfo\") pod \"7e330328-2c96-47ef-a32c-9d41b97ca03c\" (UID: \"7e330328-2c96-47ef-a32c-9d41b97ca03c\") "
Sep 16 14:05:43.542660 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542461    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:43.542660 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542481    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:43.542660 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542646    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:43.542795 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542762    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e330328-2c96-47ef-a32c-9d41b97ca03c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:05:43.542795 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542769    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.542982 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542918    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.543023 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.542988    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.543023 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.543003    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.544519 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.544489    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-kube-api-access-kwzqs" (OuterVolumeSpecName: "kube-api-access-kwzqs") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "kube-api-access-kwzqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:43.544872 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.544827    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:05:43.544966 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.544898    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:05:43.544966 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.544911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e330328-2c96-47ef-a32c-9d41b97ca03c" (UID: "7e330328-2c96-47ef-a32c-9d41b97ca03c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:43.644240 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.644199    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.644240 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.644232    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.644240 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.644242    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwzqs\" (UniqueName: \"kubernetes.io/projected/7e330328-2c96-47ef-a32c-9d41b97ca03c-kube-api-access-kwzqs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.644486 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.644251    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e330328-2c96-47ef-a32c-9d41b97ca03c-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:43.644486 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:43.644262    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e330328-2c96-47ef-a32c-9d41b97ca03c-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:44.161085 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.161053    2556 generic.go:356] "Generic (PLEG): container finished" podID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerID="cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a" exitCode=0
Sep 16 14:05:44.161287 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.161125    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"
Sep 16 14:05:44.161287 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.161131    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" event={"ID":"7e330328-2c96-47ef-a32c-9d41b97ca03c","Type":"ContainerDied","Data":"cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a"}
Sep 16 14:05:44.161287 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.161182    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz" event={"ID":"7e330328-2c96-47ef-a32c-9d41b97ca03c","Type":"ContainerDied","Data":"f239bdbec39f6435f49bd163588bf933c41178de32e71ba3d36f5ce9ee1e81ae"}
Sep 16 14:05:44.161287 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.161212    2556 scope.go:117] "RemoveContainer" containerID="cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a"
Sep 16 14:05:44.170899 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.170881    2556 scope.go:117] "RemoveContainer" containerID="a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"
Sep 16 14:05:44.178411 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.178386    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"]
Sep 16 14:05:44.180493 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.180470    2556 scope.go:117] "RemoveContainer" containerID="cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a"
Sep 16 14:05:44.180815 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:05:44.180797    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a\": container with ID starting with cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a not found: ID does not exist" containerID="cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a"
Sep 16 14:05:44.180892 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.180825    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a"} err="failed to get container status \"cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a\": rpc error: code = NotFound desc = could not find container \"cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a\": container with ID starting with cf5f279bacbd2ca649f0667c5e44eda25c32c5b83906fecc1b87eb6d778dcb3a not found: ID does not exist"
Sep 16 14:05:44.180892 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.180842    2556 scope.go:117] "RemoveContainer" containerID="a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"
Sep 16 14:05:44.181138 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:05:44.181118    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552\": container with ID starting with a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552 not found: ID does not exist" containerID="a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"
Sep 16 14:05:44.181209 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.181143    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552"} err="failed to get container status \"a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552\": rpc error: code = NotFound desc = could not find container \"a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552\": container with ID starting with a46790928e9e6a9534f1bc761d3c363f7643f03ae8553a719e76a7e8cc3a4552 not found: ID does not exist"
Sep 16 14:05:44.182136 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:44.182079    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ed470-openshift-default-6b56ccc9c4-c85vz"]
Sep 16 14:05:46.060050 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:46.060013    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" path="/var/lib/kubelet/pods/7e330328-2c96-47ef-a32c-9d41b97ca03c/volumes"
Sep 16 14:05:49.056550 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.056517    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"]
Sep 16 14:05:49.057204 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.057068    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk" podUID="a1e8556b-2b30-4416-bb5e-f0f4af8790d8" containerName="payload-processing" containerID="cri-o://9e9c7a8338a56e72b8e20651f401e58083799092681d8027062075e4db0506e7" gracePeriod=30
Sep 16 14:05:49.140305 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.140269    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:49.202317 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.202289    2556 generic.go:356] "Generic (PLEG): container finished" podID="a1e8556b-2b30-4416-bb5e-f0f4af8790d8" containerID="9e9c7a8338a56e72b8e20651f401e58083799092681d8027062075e4db0506e7" exitCode=0
Sep 16 14:05:49.202509 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.202325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk" event={"ID":"a1e8556b-2b30-4416-bb5e-f0f4af8790d8","Type":"ContainerDied","Data":"9e9c7a8338a56e72b8e20651f401e58083799092681d8027062075e4db0506e7"}
Sep 16 14:05:49.237453 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.237412    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:49.295253 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.295195    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:49.324289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.324257    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:05:49.402016 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.401978    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqjbk\" (UniqueName: \"kubernetes.io/projected/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-kube-api-access-sqjbk\") pod \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") "
Sep 16 14:05:49.402289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.402257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-plugins-config-volume\") pod \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\" (UID: \"a1e8556b-2b30-4416-bb5e-f0f4af8790d8\") "
Sep 16 14:05:49.403027 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.402988    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1e8556b-2b30-4416-bb5e-f0f4af8790d8" (UID: "a1e8556b-2b30-4416-bb5e-f0f4af8790d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:05:49.419965 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.419914    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-kube-api-access-sqjbk" (OuterVolumeSpecName: "kube-api-access-sqjbk") pod "a1e8556b-2b30-4416-bb5e-f0f4af8790d8" (UID: "a1e8556b-2b30-4416-bb5e-f0f4af8790d8"). InnerVolumeSpecName "kube-api-access-sqjbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:05:49.504405 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.504373    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-plugins-config-volume\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.504405 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.504403    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjbk\" (UniqueName: \"kubernetes.io/projected/a1e8556b-2b30-4416-bb5e-f0f4af8790d8-kube-api-access-sqjbk\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:05:49.681786 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.681735    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:49.779480 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:49.779446    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:50.210193 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:50.210160    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"
Sep 16 14:05:50.210538 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:50.210160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk" event={"ID":"a1e8556b-2b30-4416-bb5e-f0f4af8790d8","Type":"ContainerDied","Data":"f04f9396d576c6e136103842ff1e86b4a6ce11b5a348c7e7a3ae0239ce1a2034"}
Sep 16 14:05:50.211213 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:50.210296    2556 scope.go:117] "RemoveContainer" containerID="9e9c7a8338a56e72b8e20651f401e58083799092681d8027062075e4db0506e7"
Sep 16 14:05:50.226249 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:50.226220    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"]
Sep 16 14:05:50.230371 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:50.230345    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9e8244-7f7ddb46c9-6fbdk"]
Sep 16 14:05:50.379700 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:50.379612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:51.356060 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:51.356021    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:52.059235 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:52.059201    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e8556b-2b30-4416-bb5e-f0f4af8790d8" path="/var/lib/kubelet/pods/a1e8556b-2b30-4416-bb5e-f0f4af8790d8/volumes"
Sep 16 14:05:52.331928 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:52.331841    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:05:55.216818 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:55.216782    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:55.278557 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:55.278517    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:55.481200 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:55.481105    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"]
Sep 16 14:05:55.481359 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:55.481328    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy" containerID="cri-o://81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c" gracePeriod=30
Sep 16 14:05:58.183715 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:58.183683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:58.283576 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:58.283546    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:05:58.386632 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:05:58.386600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:00.744488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.744423    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:06:00.918990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.918949    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmzl9\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-kube-api-access-lmzl9\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.918990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919001    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-socket\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919029    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-istio-token\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919055    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06783f48-4ceb-4601-af37-34da7f2d593c-istiod-ca-cert\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919156    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-envoy\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919203    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06783f48-4ceb-4601-af37-34da7f2d593c-istio-podinfo\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919233    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-data\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919303 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919300    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-certs\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919616 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919355    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-credential-socket\") pod \"06783f48-4ceb-4601-af37-34da7f2d593c\" (UID: \"06783f48-4ceb-4601-af37-34da7f2d593c\") "
Sep 16 14:06:00.919616 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919406    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:00.919717 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919640    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-data" (OuterVolumeSpecName: "istio-data") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:00.919717 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919655    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:00.919894 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919875    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:00.919894 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919884    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:00.919894 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919898    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:00.919894 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919913    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:00.919894 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.919941    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06783f48-4ceb-4601-af37-34da7f2d593c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:06:00.921540 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.921513    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-kube-api-access-lmzl9" (OuterVolumeSpecName: "kube-api-access-lmzl9") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "kube-api-access-lmzl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:00.921680 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.921650    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06783f48-4ceb-4601-af37-34da7f2d593c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:06:00.921959 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.921932    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:00.921959 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:00.921944    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-istio-token" (OuterVolumeSpecName: "istio-token") pod "06783f48-4ceb-4601-af37-34da7f2d593c" (UID: "06783f48-4ceb-4601-af37-34da7f2d593c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:01.020817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.020725    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:01.020817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.020755    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmzl9\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-kube-api-access-lmzl9\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:01.020817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.020765    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06783f48-4ceb-4601-af37-34da7f2d593c-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:01.020817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.020775    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06783f48-4ceb-4601-af37-34da7f2d593c-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:01.020817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.020783    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06783f48-4ceb-4601-af37-34da7f2d593c-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:01.020817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.020790    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06783f48-4ceb-4601-af37-34da7f2d593c-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:01.280182 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.280067    2556 generic.go:356] "Generic (PLEG): container finished" podID="06783f48-4ceb-4601-af37-34da7f2d593c" containerID="81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c" exitCode=0
Sep 16 14:06:01.280182 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.280122    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" event={"ID":"06783f48-4ceb-4601-af37-34da7f2d593c","Type":"ContainerDied","Data":"81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c"}
Sep 16 14:06:01.280182 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.280138    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"
Sep 16 14:06:01.280447 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.280185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx" event={"ID":"06783f48-4ceb-4601-af37-34da7f2d593c","Type":"ContainerDied","Data":"acadb6712d328bfcfa74a795b218b3330df9cca92897b720bd6a9da4033863a2"}
Sep 16 14:06:01.280447 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.280215    2556 scope.go:117] "RemoveContainer" containerID="81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c"
Sep 16 14:06:01.298642 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.298620    2556 scope.go:117] "RemoveContainer" containerID="b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"
Sep 16 14:06:01.309934 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.309911    2556 scope.go:117] "RemoveContainer" containerID="81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c"
Sep 16 14:06:01.310310 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:06:01.310290    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c\": container with ID starting with 81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c not found: ID does not exist" containerID="81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c"
Sep 16 14:06:01.310362 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.310320    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c"} err="failed to get container status \"81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c\": rpc error: code = NotFound desc = could not find container \"81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c\": container with ID starting with 81a397e744ba8a5e8fc552da041ca328eb42a8abd80540c66291c43881ab109c not found: ID does not exist"
Sep 16 14:06:01.310362 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.310339    2556 scope.go:117] "RemoveContainer" containerID="b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"
Sep 16 14:06:01.310632 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:06:01.310616    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e\": container with ID starting with b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e not found: ID does not exist" containerID="b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"
Sep 16 14:06:01.310692 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.310635    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e"} err="failed to get container status \"b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e\": rpc error: code = NotFound desc = could not find container \"b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e\": container with ID starting with b7fc3fa5c5e8e37e4e50c8ee82504a16986391e596d7ec1ece23b663625a243e not found: ID does not exist"
Sep 16 14:06:01.310798 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.310778    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"]
Sep 16 14:06:01.314778 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:01.314756    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9e8244-openshift-default-7fbff77588-7g4sx"]
Sep 16 14:06:02.059837 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:02.059796    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" path="/var/lib/kubelet/pods/06783f48-4ceb-4601-af37-34da7f2d593c/volumes"
Sep 16 14:06:06.331439 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:06.331381    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:06.429308 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:06.429275    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:06.515314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:06.515278    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:06.582470 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:06.582384    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:06.913576 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:06.913491    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:07.013307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:07.013257    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:07.673342 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:07.673310    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:09.242600 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:09.242569    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:06:12.794795 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:12.794767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:12.856248 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:12.856218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:13.544991 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:13.544957    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:14.272260 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:14.272222    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:14.985488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:14.985449    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:15.813377 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:15.813337    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:24.489550 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:24.489506    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:24.716644 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:24.716606    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:24.774690 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:24.774602    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:25.068949 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:25.068852    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:25.169648 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:25.169611    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:27.035395 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:27.035359    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:06:31.078731 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:31.078699    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:31.425268 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:31.425230    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"]
Sep 16 14:06:31.425556 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:31.425503    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" containerID="cri-o://461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5" gracePeriod=30
Sep 16 14:06:32.071299 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:32.071236    2556 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 14:06:32.071561 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:32.071320    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 14:06:32.182714 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:32.182680    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:32.234676 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:32.234641    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:35.939366 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:35.939327    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:06:36.703403 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.703295    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 14:06:36.760381 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760333    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-certs\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760400    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-socket\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760433    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2b0e93-9314-404f-b55e-2118d4b028eb-istiod-ca-cert\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760459    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-podinfo\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760507    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-token\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760522    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-envoy\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760586 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760553    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-credential-socket\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-data\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760626    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m526d\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-kube-api-access-m526d\") pod \"1e2b0e93-9314-404f-b55e-2118d4b028eb\" (UID: \"1e2b0e93-9314-404f-b55e-2118d4b028eb\") "
Sep 16 14:06:36.760953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760686    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:36.760953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760696    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:36.760953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:36.760953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.760915    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2b0e93-9314-404f-b55e-2118d4b028eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:06:36.761314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.761024    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-credential-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.761314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.761167    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.761314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.761184    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-workload-socket\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.761314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.761197    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e2b0e93-9314-404f-b55e-2118d4b028eb-istiod-ca-cert\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.761314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.761136    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:36.762966 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.762937    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:36.763130 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.763112    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:06:36.763314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.763296    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 14:06:36.763395 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.763316    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-kube-api-access-m526d" (OuterVolumeSpecName: "kube-api-access-m526d") pod "1e2b0e93-9314-404f-b55e-2118d4b028eb" (UID: "1e2b0e93-9314-404f-b55e-2118d4b028eb"). InnerVolumeSpecName "kube-api-access-m526d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:06:36.862276 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.862230    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-podinfo\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.862276 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.862276    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-token\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.862500 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.862291    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-envoy\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.862500 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.862303    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e2b0e93-9314-404f-b55e-2118d4b028eb-istio-data\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:36.862500 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:36.862315    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m526d\" (UniqueName: \"kubernetes.io/projected/1e2b0e93-9314-404f-b55e-2118d4b028eb-kube-api-access-m526d\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:06:37.497535 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.497501    2556 generic.go:356] "Generic (PLEG): container finished" podID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerID="461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5" exitCode=0
Sep 16 14:06:37.498169 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.497564    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"
Sep 16 14:06:37.498169 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.497599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" event={"ID":"1e2b0e93-9314-404f-b55e-2118d4b028eb","Type":"ContainerDied","Data":"461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5"}
Sep 16 14:06:37.498169 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.497653    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv" event={"ID":"1e2b0e93-9314-404f-b55e-2118d4b028eb","Type":"ContainerDied","Data":"5e690e7a9472fb44af4e77a558d1013b4972ce04f367732321ae634083b5c8f4"}
Sep 16 14:06:37.498169 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.497678    2556 scope.go:117] "RemoveContainer" containerID="461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5"
Sep 16 14:06:37.510909 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.510880    2556 scope.go:117] "RemoveContainer" containerID="35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"
Sep 16 14:06:37.520904 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.520886    2556 scope.go:117] "RemoveContainer" containerID="461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5"
Sep 16 14:06:37.521280 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:06:37.521256    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5\": container with ID starting with 461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5 not found: ID does not exist" containerID="461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5"
Sep 16 14:06:37.521372 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.521289    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5"} err="failed to get container status \"461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5\": rpc error: code = NotFound desc = could not find container \"461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5\": container with ID starting with 461c47a0ce504f64e322bae2ac22a9170c350dc5a95a496e9413ed31c7e4f3b5 not found: ID does not exist"
Sep 16 14:06:37.521372 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.521309    2556 scope.go:117] "RemoveContainer" containerID="35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"
Sep 16 14:06:37.521634 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:06:37.521616    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac\": container with ID starting with 35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac not found: ID does not exist" containerID="35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"
Sep 16 14:06:37.521703 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.521644    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac"} err="failed to get container status \"35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac\": rpc error: code = NotFound desc = could not find container \"35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac\": container with ID starting with 35763d607e518692e7ca1b95af602a9cde50cf2da06223b70fda41e151c4fcac not found: ID does not exist"
Sep 16 14:06:37.521865 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.521846    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"]
Sep 16 14:06:37.525245 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:37.525222    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ac8045-openshift-default-6f69dd7d6d-nf9gv"]
Sep 16 14:06:38.060873 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:38.060837    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" path="/var/lib/kubelet/pods/1e2b0e93-9314-404f-b55e-2118d4b028eb/volumes"
Sep 16 14:06:38.198278 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:38.198249    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:50.227238 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:50.227190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:06:51.482100 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:06:51.482064    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:23.168731 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:07:23.168681    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:23.219375 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:07:23.219330    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:07:58.833073 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:07:58.833034    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:07:59.831394 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:07:59.831356    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:08:01.243974 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:08:01.243938    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:08:13.299211 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:08:13.299177    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:08:25.440851 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:08:25.440810    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:08:33.043126 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:08:33.043072    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:08:41.447247 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:08:41.447212    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:08:59.543707 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:08:59.543673    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:09:06.234901 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:09:06.234817    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:09:32.419903 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:09:32.419869    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:09:42.047590 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:09:42.047554    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:10:36.288432 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:36.288354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:10:40.266902 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:40.266860    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:10:46.336385 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:46.336350    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:10:55.244710 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.244675    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"]
Sep 16 14:10:55.245307 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245287    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245311    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245332    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e8556b-2b30-4416-bb5e-f0f4af8790d8" containerName="payload-processing"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245341    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e8556b-2b30-4416-bb5e-f0f4af8790d8" containerName="payload-processing"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245352    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245361    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245370    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" containerName="main"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245378    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" containerName="main"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245389    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245405    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245417    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy"
Sep 16 14:10:55.245423 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245425    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245438    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245445    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245460    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245468    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245490    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245498    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245611    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245624    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245635    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3324e6-8396-4d29-8bc5-3ef82d7ca6f0" containerName="main"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245643    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245653    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245661    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e8556b-2b30-4416-bb5e-f0f4af8790d8" containerName="payload-processing"
Sep 16 14:10:55.245953 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.245675    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f4c01f-7b88-4057-a5e2-c635fc7d621d" containerName="istio-proxy"
Sep 16 14:10:55.249125 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.249078    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.251062 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.251040    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-10275f33-kserve-self-signed-certs\""
Sep 16 14:10:55.256064 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.256041    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"]
Sep 16 14:10:55.303147 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.303085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-dshm\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.303293 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.303169    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvsv\" (UniqueName: \"kubernetes.io/projected/84741333-2cf2-4a45-a020-585664fecf9e-kube-api-access-zzvsv\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.303293 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.303202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84741333-2cf2-4a45-a020-585664fecf9e-tls-certs\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.303293 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.303255    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-home\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.303293 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.303278    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-model-cache\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.303479 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.303301    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-tmp-dir\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.353867 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.353836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:10:55.404518 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.404488    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-dshm\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.404708 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.404526    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvsv\" (UniqueName: \"kubernetes.io/projected/84741333-2cf2-4a45-a020-585664fecf9e-kube-api-access-zzvsv\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.404708 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.404547    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84741333-2cf2-4a45-a020-585664fecf9e-tls-certs\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.404708 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.404582    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-home\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.404708 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.404608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-model-cache\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.404708 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.404644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-tmp-dir\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.405048 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.405022    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-home\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.405157 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.405036    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-tmp-dir\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.405157 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.405109    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-model-cache\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.406904 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.406880    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-dshm\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.407249 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.407235    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84741333-2cf2-4a45-a020-585664fecf9e-tls-certs\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.412026 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.412008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvsv\" (UniqueName: \"kubernetes.io/projected/84741333-2cf2-4a45-a020-585664fecf9e-kube-api-access-zzvsv\") pod \"e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") " pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.559357 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.559260    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:10:55.831890 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.831857    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"]
Sep 16 14:10:55.833453 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:10:55.833421    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84741333_2cf2_4a45_a020_585664fecf9e.slice/crio-62b16971c06fb0063d999d8e4a5acb46ddc37d89cb732179f3f3444418d73f69 WatchSource:0}: Error finding container 62b16971c06fb0063d999d8e4a5acb46ddc37d89cb732179f3f3444418d73f69: Status 404 returned error can't find the container with id 62b16971c06fb0063d999d8e4a5acb46ddc37d89cb732179f3f3444418d73f69
Sep 16 14:10:55.984845 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.984813    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm" event={"ID":"84741333-2cf2-4a45-a020-585664fecf9e","Type":"ContainerStarted","Data":"0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15"}
Sep 16 14:10:55.985015 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:55.984854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm" event={"ID":"84741333-2cf2-4a45-a020-585664fecf9e","Type":"ContainerStarted","Data":"62b16971c06fb0063d999d8e4a5acb46ddc37d89cb732179f3f3444418d73f69"}
Sep 16 14:10:56.000498 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:56.000444    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm" podStartSLOduration=1.000426966 podStartE2EDuration="1.000426966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:10:55.998346485 +0000 UTC m=+2266.464283394" watchObservedRunningTime="2026-09-16 14:10:56.000426966 +0000 UTC m=+2266.466363874"
Sep 16 14:10:57.831761 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:10:57.831723    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:11:05.559497 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:05.559458    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:11:05.559880 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:05.559517    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:11:05.571935 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:05.571912    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:11:06.059609 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:06.059582    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:11:07.177481 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:07.177447    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:11:09.938978 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:09.938940    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:11:37.367284 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:37.367249    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"]
Sep 16 14:11:37.659079 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:37.658991    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:11:37.846608 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:37.846578    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:11:40.541239 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:40.541204    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:11:50.035146 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:50.035108    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:11:52.387504 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.387464    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm" podUID="84741333-2cf2-4a45-a020-585664fecf9e" containerName="main" containerID="cri-o://0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15" gracePeriod=45
Sep 16 14:11:52.641370 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.641311    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:11:52.761270 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761237    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-home\") pod \"84741333-2cf2-4a45-a020-585664fecf9e\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") "
Sep 16 14:11:52.761488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761277    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-tmp-dir\") pod \"84741333-2cf2-4a45-a020-585664fecf9e\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") "
Sep 16 14:11:52.761488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761307    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-dshm\") pod \"84741333-2cf2-4a45-a020-585664fecf9e\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") "
Sep 16 14:11:52.761488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761349    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzvsv\" (UniqueName: \"kubernetes.io/projected/84741333-2cf2-4a45-a020-585664fecf9e-kube-api-access-zzvsv\") pod \"84741333-2cf2-4a45-a020-585664fecf9e\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") "
Sep 16 14:11:52.761488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761385    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-model-cache\") pod \"84741333-2cf2-4a45-a020-585664fecf9e\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") "
Sep 16 14:11:52.761488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761445    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84741333-2cf2-4a45-a020-585664fecf9e-tls-certs\") pod \"84741333-2cf2-4a45-a020-585664fecf9e\" (UID: \"84741333-2cf2-4a45-a020-585664fecf9e\") "
Sep 16 14:11:52.761763 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761563    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84741333-2cf2-4a45-a020-585664fecf9e" (UID: "84741333-2cf2-4a45-a020-585664fecf9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:52.761763 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761597    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-home" (OuterVolumeSpecName: "home") pod "84741333-2cf2-4a45-a020-585664fecf9e" (UID: "84741333-2cf2-4a45-a020-585664fecf9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:52.761763 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761715    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "84741333-2cf2-4a45-a020-585664fecf9e" (UID: "84741333-2cf2-4a45-a020-585664fecf9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:52.761911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761829    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-home\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:11:52.761911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761850    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-tmp-dir\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:11:52.761911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.761868    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-model-cache\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:11:52.763745 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.763715    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-dshm" (OuterVolumeSpecName: "dshm") pod "84741333-2cf2-4a45-a020-585664fecf9e" (UID: "84741333-2cf2-4a45-a020-585664fecf9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 14:11:52.763745 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.763740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84741333-2cf2-4a45-a020-585664fecf9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84741333-2cf2-4a45-a020-585664fecf9e" (UID: "84741333-2cf2-4a45-a020-585664fecf9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:11:52.763891 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.763740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84741333-2cf2-4a45-a020-585664fecf9e-kube-api-access-zzvsv" (OuterVolumeSpecName: "kube-api-access-zzvsv") pod "84741333-2cf2-4a45-a020-585664fecf9e" (UID: "84741333-2cf2-4a45-a020-585664fecf9e"). InnerVolumeSpecName "kube-api-access-zzvsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:11:52.862409 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.862377    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84741333-2cf2-4a45-a020-585664fecf9e-dshm\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:11:52.862409 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.862406    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzvsv\" (UniqueName: \"kubernetes.io/projected/84741333-2cf2-4a45-a020-585664fecf9e-kube-api-access-zzvsv\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:11:52.862409 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:52.862417    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84741333-2cf2-4a45-a020-585664fecf9e-tls-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:11:53.317714 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.317679    2556 generic.go:356] "Generic (PLEG): container finished" podID="84741333-2cf2-4a45-a020-585664fecf9e" containerID="0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15" exitCode=0
Sep 16 14:11:53.317911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.317726    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"
Sep 16 14:11:53.317911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.317726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm" event={"ID":"84741333-2cf2-4a45-a020-585664fecf9e","Type":"ContainerDied","Data":"0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15"}
Sep 16 14:11:53.317911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.317847    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm" event={"ID":"84741333-2cf2-4a45-a020-585664fecf9e","Type":"ContainerDied","Data":"62b16971c06fb0063d999d8e4a5acb46ddc37d89cb732179f3f3444418d73f69"}
Sep 16 14:11:53.317911 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.317871    2556 scope.go:117] "RemoveContainer" containerID="0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15"
Sep 16 14:11:53.328455 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.328435    2556 scope.go:117] "RemoveContainer" containerID="0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15"
Sep 16 14:11:53.328751 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:11:53.328734    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15\": container with ID starting with 0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15 not found: ID does not exist" containerID="0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15"
Sep 16 14:11:53.328803 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.328760    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15"} err="failed to get container status \"0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15\": rpc error: code = NotFound desc = could not find container \"0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15\": container with ID starting with 0817bd5cf0415a9c0a80d89957c8bec0e5ad5d1f64b88d14cda15de409b69a15 not found: ID does not exist"
Sep 16 14:11:53.339478 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.339454    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"]
Sep 16 14:11:53.342942 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:53.342918    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-10275f33-kserve-79b7588f5f-b49lm"]
Sep 16 14:11:54.058970 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:54.058935    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84741333-2cf2-4a45-a020-585664fecf9e" path="/var/lib/kubelet/pods/84741333-2cf2-4a45-a020-585664fecf9e/volumes"
Sep 16 14:11:56.833122 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:56.833077    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:11:57.284311 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284279    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"]
Sep 16 14:11:57.284761 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284743    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84741333-2cf2-4a45-a020-585664fecf9e" containerName="main"
Sep 16 14:11:57.284853 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284764    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="84741333-2cf2-4a45-a020-585664fecf9e" containerName="main"
Sep 16 14:11:57.284853 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284785    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy"
Sep 16 14:11:57.284853 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284796    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy"
Sep 16 14:11:57.285004 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284903    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="84741333-2cf2-4a45-a020-585664fecf9e" containerName="main"
Sep 16 14:11:57.285004 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284916    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e2b0e93-9314-404f-b55e-2118d4b028eb" containerName="istio-proxy"
Sep 16 14:11:57.285004 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284934    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e330328-2c96-47ef-a32c-9d41b97ca03c" containerName="istio-proxy"
Sep 16 14:11:57.285004 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.284945    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="06783f48-4ceb-4601-af37-34da7f2d593c" containerName="istio-proxy"
Sep 16 14:11:57.288129 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.288109    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.299142 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.299118    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"]
Sep 16 14:11:57.299915 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.299895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-kube-api-access-7fhvv\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.299990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.299929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-maas-api-tls\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.299990 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.299974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-metrics-certs\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.300074 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.300007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-openshift-service-ca\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.401385 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.401352    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-maas-api-tls\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.401600 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.401413    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-metrics-certs\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.401600 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.401449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-openshift-service-ca\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.401600 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.401510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-kube-api-access-7fhvv\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.402181 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.402120    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-openshift-service-ca\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.404046 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.404019    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-maas-api-tls\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.404164 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.404068    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-metrics-certs\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.408597 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.408579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fhvv\" (UniqueName: \"kubernetes.io/projected/dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2-kube-api-access-7fhvv\") pod \"maas-api-85fc744486-bkbpl\" (UID: \"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2\") " pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.599786 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.599693    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:11:57.871633 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.871607    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"]
Sep 16 14:11:57.873137 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:11:57.873105    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4a5743_3cf4_4a48_b16f_cf2eaa9a76d2.slice/crio-b1c32ea3ade81bdc65fc3bbc4b0ccc4a5017e4b4e3a77ff55ebf35c8eaa80351 WatchSource:0}: Error finding container b1c32ea3ade81bdc65fc3bbc4b0ccc4a5017e4b4e3a77ff55ebf35c8eaa80351: Status 404 returned error can't find the container with id b1c32ea3ade81bdc65fc3bbc4b0ccc4a5017e4b4e3a77ff55ebf35c8eaa80351
Sep 16 14:11:57.874504 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:57.874482    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 14:11:58.348755 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:11:58.348719    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl" event={"ID":"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2","Type":"ContainerStarted","Data":"b1c32ea3ade81bdc65fc3bbc4b0ccc4a5017e4b4e3a77ff55ebf35c8eaa80351"}
Sep 16 14:12:00.362480 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:00.362442    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl" event={"ID":"dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2","Type":"ContainerStarted","Data":"657f1d187129f7b4314639eacc29e02faeb6e824f7e3015d92cbccd1c3a5d37d"}
Sep 16 14:12:00.362965 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:00.362576    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:12:00.381983 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:00.381931    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl" podStartSLOduration=1.5614945310000001 podStartE2EDuration="3.381916404s" totalImagesPullingTime="1.820421873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 14:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:00.379332995 +0000 UTC m=+2330.845269931" watchObservedRunningTime="2026-09-16 14:12:00.381916404 +0000 UTC m=+2330.847853313"
Sep 16 14:12:04.929242 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:04.929148    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:12:06.374003 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:06.373973    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85fc744486-bkbpl"
Sep 16 14:12:06.419111 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:06.419053    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"]
Sep 16 14:12:06.419409 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:06.419350    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" podUID="716f3741-ad4a-4d42-9bbc-54b281b53085" containerName="maas-api" containerID="cri-o://33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05" gracePeriod=30
Sep 16 14:12:07.039391 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.039340    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" podUID="716f3741-ad4a-4d42-9bbc-54b281b53085" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.52:8443/health\": dial tcp 10.133.0.52:8443: connect: connection refused"
Sep 16 14:12:07.676114 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.676076    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 14:12:07.799286 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.799205    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-metrics-certs\") pod \"716f3741-ad4a-4d42-9bbc-54b281b53085\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") "
Sep 16 14:12:07.799286 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.799240    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716f3741-ad4a-4d42-9bbc-54b281b53085-openshift-service-ca\") pod \"716f3741-ad4a-4d42-9bbc-54b281b53085\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") "
Sep 16 14:12:07.799286 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.799287    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwhjx\" (UniqueName: \"kubernetes.io/projected/716f3741-ad4a-4d42-9bbc-54b281b53085-kube-api-access-pwhjx\") pod \"716f3741-ad4a-4d42-9bbc-54b281b53085\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") "
Sep 16 14:12:07.799513 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.799332    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-maas-api-tls\") pod \"716f3741-ad4a-4d42-9bbc-54b281b53085\" (UID: \"716f3741-ad4a-4d42-9bbc-54b281b53085\") "
Sep 16 14:12:07.799735 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.799711    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716f3741-ad4a-4d42-9bbc-54b281b53085-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "716f3741-ad4a-4d42-9bbc-54b281b53085" (UID: "716f3741-ad4a-4d42-9bbc-54b281b53085"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 14:12:07.801709 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.801682    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716f3741-ad4a-4d42-9bbc-54b281b53085-kube-api-access-pwhjx" (OuterVolumeSpecName: "kube-api-access-pwhjx") pod "716f3741-ad4a-4d42-9bbc-54b281b53085" (UID: "716f3741-ad4a-4d42-9bbc-54b281b53085"). InnerVolumeSpecName "kube-api-access-pwhjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 14:12:07.801709 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.801696    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "716f3741-ad4a-4d42-9bbc-54b281b53085" (UID: "716f3741-ad4a-4d42-9bbc-54b281b53085"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:12:07.801852 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.801716    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "716f3741-ad4a-4d42-9bbc-54b281b53085" (UID: "716f3741-ad4a-4d42-9bbc-54b281b53085"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 14:12:07.900628 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.900593    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-metrics-certs\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:12:07.900628 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.900621    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/716f3741-ad4a-4d42-9bbc-54b281b53085-openshift-service-ca\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:12:07.900628 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.900630    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwhjx\" (UniqueName: \"kubernetes.io/projected/716f3741-ad4a-4d42-9bbc-54b281b53085-kube-api-access-pwhjx\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:12:07.900628 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:07.900639    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/716f3741-ad4a-4d42-9bbc-54b281b53085-maas-api-tls\") on node \"ip-10-0-133-251.ec2.internal\" DevicePath \"\""
Sep 16 14:12:08.411658 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.411568    2556 generic.go:356] "Generic (PLEG): container finished" podID="716f3741-ad4a-4d42-9bbc-54b281b53085" containerID="33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05" exitCode=0
Sep 16 14:12:08.411658 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.411607    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" event={"ID":"716f3741-ad4a-4d42-9bbc-54b281b53085","Type":"ContainerDied","Data":"33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05"}
Sep 16 14:12:08.411658 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.411655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft" event={"ID":"716f3741-ad4a-4d42-9bbc-54b281b53085","Type":"ContainerDied","Data":"050b86411a4ff0195b35fff91af9bb46d9d98f8cbc554d29f934507962b44f3f"}
Sep 16 14:12:08.411935 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.411684    2556 scope.go:117] "RemoveContainer" containerID="33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05"
Sep 16 14:12:08.411935 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.411609    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"
Sep 16 14:12:08.420856 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.420839    2556 scope.go:117] "RemoveContainer" containerID="33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05"
Sep 16 14:12:08.421160 ip-10-0-133-251 kubenswrapper[2556]: E0916 14:12:08.421141    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05\": container with ID starting with 33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05 not found: ID does not exist" containerID="33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05"
Sep 16 14:12:08.421218 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.421170    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05"} err="failed to get container status \"33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05\": rpc error: code = NotFound desc = could not find container \"33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05\": container with ID starting with 33e4e7f164038e1524c29002a91d97dc9a4ecbe12c4f6cec428604c22999fa05 not found: ID does not exist"
Sep 16 14:12:08.426011 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.425990    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"]
Sep 16 14:12:08.429787 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:08.429765    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-67c45f647d-6nwft"]
Sep 16 14:12:10.058883 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:10.058841    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="716f3741-ad4a-4d42-9bbc-54b281b53085" path="/var/lib/kubelet/pods/716f3741-ad4a-4d42-9bbc-54b281b53085/volumes"
Sep 16 14:12:11.737075 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:11.737041    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:12:17.630902 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:17.630869    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:12:24.828697 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:24.828658    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-47spw"]
Sep 16 14:12:32.937357 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:32.937328    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kl2sd_0c15b622-2c5b-4be7-8588-22fc547cc1ad/manager/0.log"
Sep 16 14:12:33.062366 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:33.062313    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vgmpj_932290b0-d3d3-4367-a0c0-4a980bd5b55e/manager/0.log"
Sep 16 14:12:33.547349 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:33.547317    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6444576fdb-54p72_ebe934fe-558a-4c8a-82c0-339af02098f8/copy-manifests/0.log"
Sep 16 14:12:33.552799 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:33.552773    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6444576fdb-54p72_ebe934fe-558a-4c8a-82c0-339af02098f8/manager/0.log"
Sep 16 14:12:34.307685 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:34.307652    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85fc744486-bkbpl_dd4a5743-3cf4-4a48-b16f-cf2eaa9a76d2/maas-api/0.log"
Sep 16 14:12:34.536449 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:34.536421    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826120-rkssb_af2c2a83-9ea8-4d11-abc5-bb95dcd72da8/cleanup/0.log"
Sep 16 14:12:34.652917 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:34.652829    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-wk4hb_82847242-d02e-4187-8ffd-74960e4806d4/postgres/0.log"
Sep 16 14:12:35.561067 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:35.561036    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-886x4_e5ca2022-9943-4720-b501-d802020e8b4e/manager/0.log"
Sep 16 14:12:35.670782 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:35.670747    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-46n6f_33075a5f-302c-4eb1-a901-83cab15744d4/manager/0.log"
Sep 16 14:12:36.024926 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:36.024901    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-47spw_ebfc6e40-8d5b-402f-82cb-da81f6bd4343/limitador/0.log"
Sep 16 14:12:36.145223 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:36.145194    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kc4n6_ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b/manager/0.log"
Sep 16 14:12:36.771072 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:36.771040    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68bbd697f4-67sml_d8e2d3bb-6b44-4da3-b1a9-f515693da377/kube-auth-proxy/0.log"
Sep 16 14:12:37.003610 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:37.003581    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/1.log"
Sep 16 14:12:37.250203 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:37.250178    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6j2c_ed380fe8-2bf2-4ad0-968e-341cbb8a3e17/payload-processing/0.log"
Sep 16 14:12:37.369151 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:37.369119    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79cff5dd-mnnh2_8d999b14-0ce5-49e5-8c6e-3c74fedd210c/router/0.log"
Sep 16 14:12:38.053261 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:38.053227    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6btvw_02d30d6c-791a-46e9-90c9-87be00d1269e/main/0.log"
Sep 16 14:12:38.381322 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:38.381243    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-lbdkq_1f5c957b-52ab-4eda-97e0-51435c43e3c2/main/0.log"
Sep 16 14:12:43.566005 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:43.565960    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:12:43.628900 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:43.628866    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log" limit=1024
Sep 16 14:12:46.304065 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:46.304036    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cxlj6_54d8f078-4001-4193-8c85-2951ebc88278/global-pull-secret-syncer/0.log"
Sep 16 14:12:46.420499 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:46.420467    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rvzwh_f7fb372c-6db1-43f6-9e4d-094f9cf7b9e9/konnectivity-agent/0.log"
Sep 16 14:12:46.547042 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:46.546992    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-251.ec2.internal_7dc59d1cdb7714ddbcf795e023ef2c6a/haproxy/0.log"
Sep 16 14:12:51.600253 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:51.600218    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-886x4_e5ca2022-9943-4720-b501-d802020e8b4e/manager/0.log"
Sep 16 14:12:51.625766 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:51.625736    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-46n6f_33075a5f-302c-4eb1-a901-83cab15744d4/manager/0.log"
Sep 16 14:12:51.713380 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:51.713351    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-47spw_ebfc6e40-8d5b-402f-82cb-da81f6bd4343/limitador/0.log"
Sep 16 14:12:51.776187 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:51.776152    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kc4n6_ff61ebd2-8da1-4fe4-98c6-261fd4c0c31b/manager/0.log"
Sep 16 14:12:53.262082 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.262046    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-bzw76_68537ef5-22fb-4534-9627-5441ca8ac4a5/cluster-monitoring-operator/0.log"
Sep 16 14:12:53.286610 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.286586    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lm26r_3258470a-1466-40e0-afca-08530a9a88e9/kube-state-metrics/0.log"
Sep 16 14:12:53.306424 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.306401    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lm26r_3258470a-1466-40e0-afca-08530a9a88e9/kube-rbac-proxy-main/0.log"
Sep 16 14:12:53.326668 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.326640    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-lm26r_3258470a-1466-40e0-afca-08530a9a88e9/kube-rbac-proxy-self/0.log"
Sep 16 14:12:53.413074 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.413044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nqnw_ebbce715-10b6-4573-acde-12b63cd9beb4/node-exporter/0.log"
Sep 16 14:12:53.432442 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.432421    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nqnw_ebbce715-10b6-4573-acde-12b63cd9beb4/kube-rbac-proxy/0.log"
Sep 16 14:12:53.453634 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:53.453614    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2nqnw_ebbce715-10b6-4573-acde-12b63cd9beb4/init-textfile/0.log"
Sep 16 14:12:54.086214 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.086135    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79cd7db795-9hmgk_b2852957-d8f2-4947-8eac-b9227c0110bc/telemeter-client/0.log"
Sep 16 14:12:54.109549 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.109523    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79cd7db795-9hmgk_b2852957-d8f2-4947-8eac-b9227c0110bc/reload/0.log"
Sep 16 14:12:54.130376 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.130350    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79cd7db795-9hmgk_b2852957-d8f2-4947-8eac-b9227c0110bc/kube-rbac-proxy/0.log"
Sep 16 14:12:54.164262 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.164234    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f89c58984-jfr88_422ce74c-8838-43fa-91db-1a29dae00fcb/thanos-query/0.log"
Sep 16 14:12:54.185873 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.185847    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f89c58984-jfr88_422ce74c-8838-43fa-91db-1a29dae00fcb/kube-rbac-proxy-web/0.log"
Sep 16 14:12:54.206695 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.206670    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f89c58984-jfr88_422ce74c-8838-43fa-91db-1a29dae00fcb/kube-rbac-proxy/0.log"
Sep 16 14:12:54.228761 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.228734    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f89c58984-jfr88_422ce74c-8838-43fa-91db-1a29dae00fcb/prom-label-proxy/0.log"
Sep 16 14:12:54.246028 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.246002    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f89c58984-jfr88_422ce74c-8838-43fa-91db-1a29dae00fcb/kube-rbac-proxy-rules/0.log"
Sep 16 14:12:54.266139 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:54.266121    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f89c58984-jfr88_422ce74c-8838-43fa-91db-1a29dae00fcb/kube-rbac-proxy-metrics/0.log"
Sep 16 14:12:55.633451 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:55.633418    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-lngmg_62b3e5d8-9b13-49cc-98ad-b058ac68c5f7/networking-console-plugin/0.log"
Sep 16 14:12:56.787962 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:56.787934    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5947c4777d-hpmgb_31cb106b-fb16-4ac7-a5b9-2bc9659b97be/console/0.log"
Sep 16 14:12:57.088621 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.088528    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"]
Sep 16 14:12:57.089034 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.089015    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="716f3741-ad4a-4d42-9bbc-54b281b53085" containerName="maas-api"
Sep 16 14:12:57.089159 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.089038    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="716f3741-ad4a-4d42-9bbc-54b281b53085" containerName="maas-api"
Sep 16 14:12:57.089159 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.089151    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="716f3741-ad4a-4d42-9bbc-54b281b53085" containerName="maas-api"
Sep 16 14:12:57.092288 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.092266    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.094375 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.094353    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t22r\"/\"openshift-service-ca.crt\""
Sep 16 14:12:57.095314 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.095289    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6t22r\"/\"kube-root-ca.crt\""
Sep 16 14:12:57.095480 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.095313    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6t22r\"/\"default-dockercfg-qntd4\""
Sep 16 14:12:57.102310 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.101885    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"]
Sep 16 14:12:57.168817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.168780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-podres\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.168817 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.168819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-proc\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.169078 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.168959    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-sys\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.169078 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.169007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-lib-modules\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.169228 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.169117    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjr7c\" (UniqueName: \"kubernetes.io/projected/4b4be19a-6326-4f1a-a302-b78772bf7188-kube-api-access-wjr7c\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270050 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-sys\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270050 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-lib-modules\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjr7c\" (UniqueName: \"kubernetes.io/projected/4b4be19a-6326-4f1a-a302-b78772bf7188-kube-api-access-wjr7c\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270143    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-podres\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-proc\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270167    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-sys\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-lib-modules\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270235    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-proc\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.270289 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.270268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b4be19a-6326-4f1a-a302-b78772bf7188-podres\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.278536 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.278509    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjr7c\" (UniqueName: \"kubernetes.io/projected/4b4be19a-6326-4f1a-a302-b78772bf7188-kube-api-access-wjr7c\") pod \"perf-node-gather-daemonset-slkjj\" (UID: \"4b4be19a-6326-4f1a-a302-b78772bf7188\") " pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.410954 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.410880    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:57.681721 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:57.681695    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"]
Sep 16 14:12:57.684030 ip-10-0-133-251 kubenswrapper[2556]: W0916 14:12:57.684000    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4b4be19a_6326_4f1a_a302_b78772bf7188.slice/crio-3e890b47a9b891bb23094adcb96117d118d7bf02eb5e622a65268d352d28f88e WatchSource:0}: Error finding container 3e890b47a9b891bb23094adcb96117d118d7bf02eb5e622a65268d352d28f88e: Status 404 returned error can't find the container with id 3e890b47a9b891bb23094adcb96117d118d7bf02eb5e622a65268d352d28f88e
Sep 16 14:12:58.650811 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.650780    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xcljx_27eed164-faa1-4a17-ab02-a5f6528389bb/dns/0.log"
Sep 16 14:12:58.672444 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.672418    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xcljx_27eed164-faa1-4a17-ab02-a5f6528389bb/kube-rbac-proxy/0.log"
Sep 16 14:12:58.694604 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.694566    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj" event={"ID":"4b4be19a-6326-4f1a-a302-b78772bf7188","Type":"ContainerStarted","Data":"5ae133680c9f97f8e564e8b72195fb649502f59c0c5b4f1f43dc5f71ef2fc5c6"}
Sep 16 14:12:58.694759 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.694623    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj" event={"ID":"4b4be19a-6326-4f1a-a302-b78772bf7188","Type":"ContainerStarted","Data":"3e890b47a9b891bb23094adcb96117d118d7bf02eb5e622a65268d352d28f88e"}
Sep 16 14:12:58.694759 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.694648    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:12:58.710960 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.710918    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj" podStartSLOduration=1.71090395 podStartE2EDuration="1.71090395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 14:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 14:12:58.708229073 +0000 UTC m=+2389.174166002" watchObservedRunningTime="2026-09-16 14:12:58.71090395 +0000 UTC m=+2389.176840872"
Sep 16 14:12:58.742184 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:58.742159    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52d77_27850248-ce2d-480c-83f7-05c2477fb14f/dns-node-resolver/0.log"
Sep 16 14:12:59.307071 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:59.307042    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c65897dc6-5b5f7_1a6f5ee2-d016-4e1e-9497-6a2bce11f846/registry/0.log"
Sep 16 14:12:59.374803 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:12:59.374773    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vp7zm_da85859e-800e-4d7c-8b95-775d36698fdd/node-ca/0.log"
Sep 16 14:13:00.827548 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:00.827520    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68bbd697f4-67sml_d8e2d3bb-6b44-4da3-b1a9-f515693da377/kube-auth-proxy/0.log"
Sep 16 14:13:00.882300 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:00.882271    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/0.log"
Sep 16 14:13:00.885021 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:00.885000    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/1.log"
Sep 16 14:13:00.946560 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:00.946528    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6j2c_ed380fe8-2bf2-4ad0-968e-341cbb8a3e17/payload-processing/0.log"
Sep 16 14:13:00.974545 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:00.974513    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79cff5dd-mnnh2_8d999b14-0ce5-49e5-8c6e-3c74fedd210c/router/0.log"
Sep 16 14:13:01.882631 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:01.882603    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bj9zd_fa9b128f-810f-4a59-9ce9-c74bbe571128/serve-healthcheck-canary/0.log"
Sep 16 14:13:02.373600 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:02.373570    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qcnrn_e9af322c-e51f-4ca3-ae75-ac4dc4e7da37/kube-rbac-proxy/0.log"
Sep 16 14:13:02.394523 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:02.394491    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qcnrn_e9af322c-e51f-4ca3-ae75-ac4dc4e7da37/exporter/0.log"
Sep 16 14:13:02.414888 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:02.414855    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qcnrn_e9af322c-e51f-4ca3-ae75-ac4dc4e7da37/extractor/0.log"
Sep 16 14:13:04.696654 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:04.696623    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kl2sd_0c15b622-2c5b-4be7-8588-22fc547cc1ad/manager/0.log"
Sep 16 14:13:04.710478 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:04.710453    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6t22r/perf-node-gather-daemonset-slkjj"
Sep 16 14:13:04.945905 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:04.945864    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vgmpj_932290b0-d3d3-4367-a0c0-4a980bd5b55e/manager/0.log"
Sep 16 14:13:05.140384 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:05.140300    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6444576fdb-54p72_ebe934fe-558a-4c8a-82c0-339af02098f8/manager/0.log"
Sep 16 14:13:05.159297 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:05.159272    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6444576fdb-54p72_ebe934fe-558a-4c8a-82c0-339af02098f8/copy-manifests/0.log"
Sep 16 14:13:06.976076 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:06.976049    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68bbd697f4-67sml_d8e2d3bb-6b44-4da3-b1a9-f515693da377/kube-auth-proxy/0.log"
Sep 16 14:13:07.013526 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:07.013446    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/0.log"
Sep 16 14:13:07.015984 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:07.015959    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/1.log"
Sep 16 14:13:07.060488 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:07.060458    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6j2c_ed380fe8-2bf2-4ad0-968e-341cbb8a3e17/payload-processing/0.log"
Sep 16 14:13:07.076877 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:07.076850    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79cff5dd-mnnh2_8d999b14-0ce5-49e5-8c6e-3c74fedd210c/router/0.log"
Sep 16 14:13:08.991179 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:08.991151    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68bbd697f4-67sml_d8e2d3bb-6b44-4da3-b1a9-f515693da377/kube-auth-proxy/0.log"
Sep 16 14:13:09.039603 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:09.039573    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/0.log"
Sep 16 14:13:09.042490 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:09.042470    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-df6776dd6-gkxhg_af20b64c-0972-4684-9fe0-c0f769331b52/istio-proxy/1.log"
Sep 16 14:13:09.089766 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:09.089734    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-x6j2c_ed380fe8-2bf2-4ad0-968e-341cbb8a3e17/payload-processing/0.log"
Sep 16 14:13:09.107247 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:09.107222    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b79cff5dd-mnnh2_8d999b14-0ce5-49e5-8c6e-3c74fedd210c/router/0.log"
Sep 16 14:13:12.198107 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:12.198058    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-b4lwr_d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3/kube-storage-version-migrator-operator/1.log"
Sep 16 14:13:12.198856 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:12.198834    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-b4lwr_d20ab845-bbe8-4d3f-ad4e-f22f6fd062f3/kube-storage-version-migrator-operator/0.log"
Sep 16 14:13:13.543282 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.543253    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/kube-multus-additional-cni-plugins/0.log"
Sep 16 14:13:13.562880 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.562849    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/egress-router-binary-copy/0.log"
Sep 16 14:13:13.582838 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.582806    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/cni-plugins/0.log"
Sep 16 14:13:13.602160 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.602135    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/bond-cni-plugin/0.log"
Sep 16 14:13:13.621106 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.621069    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/routeoverride-cni/0.log"
Sep 16 14:13:13.640382 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.640357    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/whereabouts-cni-bincopy/0.log"
Sep 16 14:13:13.658962 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.658941    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-x92sh_0c5a16d7-8099-4cf5-a759-1684dbbc46a0/whereabouts-cni/0.log"
Sep 16 14:13:13.700866 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.700836    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vkg5d_a32af615-2666-4782-807f-c7fa822b8706/kube-multus/0.log"
Sep 16 14:13:13.724567 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.724541    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8w2s9_2786ffeb-964f-42fe-995f-e3be60ecd47f/network-metrics-daemon/0.log"
Sep 16 14:13:13.744831 ip-10-0-133-251 kubenswrapper[2556]: I0916 14:13:13.744776    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8w2s9_2786ffeb-964f-42fe-995f-e3be60ecd47f/kube-rbac-proxy/0.log"